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ABSTRACT 

This report supplements the final evaluation of the 
State Dissemination Grants Program (SDGP) , a ma^or initiative within 
the mission of the National Institute of Education to assist state 
educational agencies in implementing, strengthening, and 
institutionalizing dissemination services that improve educational 
practice and equity. The introduction to this substudy explains that 
data collected for the final evaluation were subjected to a scaling 
analysis to: (1) confirm that indicants grouped together on the basis 
of their content were, in fact, measuring the same thing; (2) 
eliminate indicants not related to the other indicants in the facet; 
(3) parsimoniously summarize a large body of data; and (4) serve as a 
measuring device, by assigning facet scores to each scala for use in 
characterizing state capacity. The four main sections of this report 
discuss the five-component conceptual framework used to specify and 
organize the variables which define the process of building state 
dissemination capacity; the scaling procedures and the application of 
Rasch scaling analysis; the degree to which the scales provide a 
developmental perspective ^o the capacity building process; and 
interpretation and utilization of the resultant information for 
program improvement. Data are presented in 13 tables and 14 figures. 
(NRP) 
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PREFACE 



The State Dissemination Grants Program is a major initiative within the 
mission of the National Institute of Education (NIE) "to promote educational 
equity and improve the quality of educational practice." NIE expects the 
State Dissemination Grants Program (SDGP) will aid the development of a 
nationwide capability for educational improvement by assisting a significant 
group of actors, state education agencies (SEAs), to implement, strengthen, 
and institutionalize dissemination services that improve educational practice 
and equity. 

Under the sponsorship of the NIE's Program on Dissemination and Improve- 
ment of Practice, NTS Research Corporation conducted a multi-year study of 
the State Dissemination Grants Program (NIE Contract No. 400-76-0166, October 
1976 - April 1980) to address two major questions: 

• Is dissemination capacity being built as a result of this 
program? If so, how? 

• Is the program having an effect? If so, what is the nature 
of the effect? 

The evaluation was in two phases. Phase I of the study, an eieven-month 
design phase that extended from October 1976 through August 1977, was devoted 
to describing the program, clarifying and translating the program's goals 
into measurable variables, and developing a design, appropriate instrumenta- 
tion, and data collection and analysis procedures for the study. Familiari- 
zation visits to 23 projects, refinements in the study design, and approval 
of a forms clearance package occurred during September 1977 - August 1978. 
Phase II, the full-scale evaluation, was initiated in September 1978 and con- 
cluded in April 1980. Phase II objectives included describing and ^t^??.king 
the process of building dissemination capacity, documenting the impact of the 
program, sharing the study findings and analyses with NIE and the states to 
promote program and project improvement, and developing mechanisms for the 
continual evaluation and measurement of dissemination Ccpacity. 

The final report for the NTS study is comprised of four volumes: 

f This volume. Volume IV: A Study of the Development of Scales 
Measuring Dissemination Capacity (April 1981) is a technical 
report which describes how the scales were developed and how 
they have been used. 
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• Volume I: Building Capacity for Improvement of Education; 
An Evaluation of NIE's State Dissemination Grants Program 
(April 1981), is the final evaluation report of the State 
Dissemination Grants Program. Included are descriptions of 
the program and the evaluation, of qualitative cross-case 
analyses of five capacity building states, generic descrip- 
tions of state dissemination systems, and quantitative analy- 
ses which identify factors which facilitate and impede the 
development and institutionalization of SEA dissemination 
systems. The analyses reveal that dissemination capacity is 
being built, participation in the program enhances such 
growth, and SEA dissemination systems of states participat- 
ing in the program differ from those of non -participating 
states. A final chapter discusses the policy implications 

of these and other findings. 

• Volume II: 1979 State Abstracts; State Dissemination Efforts 
(April 1980), profiles dissemination activities in thirty-eight 
SEAs as of December 1979. In addition to summaries of capac- 
ity building project states, this document describes the status 
of dissemination efforts in states that have not participated 
in the program. 

• Volume III; A Study of Linker Activities and Roles (April 1981), 
describes how people help others access and use information for 
school improvement. The study is based on data collected from 
linkers associated with the program. 

Prior to 1980, seven major reports were prepared under Phase II of the 
NTS study: 

• 1978 State Abstracts (March 1979) contains summaries of dis- 
semination activities in twenty-nine SEAs as of November 
1978. Included are nine SEAs initially funded in 1975, four- 
teen additional projects initially funded in 1976, and six 
SEAs initially funded in 1977. An introductory chapter pre- 
sencs an analysis across the individual projects. 

• Buil ding Capacity for Improvement of Education; An Evalua - 
tion of NIE s State Dissemination Grants Program, Interim 
Report , (July 1979) is the interim report on the full-scale 
evaluation of the State Dissemination Grants Program. In- 
cluded is an overview of the evaluation, purposes of the 
study, framework, evaluation questions, data collection 
methods, analytic techniques, and findings. The process 
used by SEAs to develop capacity for gaining access to in- 
formation resources and for linking such resources to the 
needs of educators are described. 
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• Intelligence for Dissemination Service Capacity; A Concer > 
tual Framework ( March 1979) expands an earlier framework 
into a heuristic device for studying users of educational 
dissemination services. This conceptual framework was com- 
pleted to guide the development and refinement of questions, 
variables, and instrumentation for users and usages of dis- 
semination services. 

• Information About Users and Usages; A Literature Review 
(March 1979) is a companion document to A Conceptual Frame - 
work . The review was prepared as part of the design process 
used to develop the framework. 

t The Client Assessment Package (December 1978) is a set of 
five machine-readable instruments developed by NTS to record 
the process of seeking and using information and assistance 
for educational Improvement. Linked by a common identifica- 
tion field, the five forms in the package are the Service 
Form, Process Form, Linker Form, Immediate Feedback Form, 
and Client Assessment Form. An accompanying Guide to the 
Client Assessment Package provides instructions for complet- 
ing and using the forms. 

• Request for 0MB Clearance with Supporting Documents for the 
Ivaluation of t h e State Capacity Building Program in Dissemi- 
nation (June ijiTj) is the justification and instrumentation 
package pr^parecJ for and approved by the Office of Management 
and Budget for use in the evaluation. 

• A Framework for the Evaluation of the State Capacity Build - 
ing Program (May 1978) presents the organizing framework for 
the evaluation. 

During Phase I of the NTS study, five major documents were also pro- 
duced; 

• 1977 State Abstracts (September 1977) contains summaries of 
twenty-four capacity building projects. Included are the 
ten states initially funded in 1975 and the fourteen addi- 
tional projects funded in 1976. The abstracts document dis- 
semination activities in the SEAs as of May 1977. 

• State Reports (Jul^ ±977) contains extensive documentation 
on nine of the first states funded through the capacity 
building component of the State Dissemination Grants Program. 
The mini-case studies xamine dissemination activities in 
nine SEAs as of May 1977. 

• A Compendium of Evaluation and Documentation Forms Currently 
Tn Use by State Capacity Bui Iding Projects (July 1977) is a 
compilation of selected instrumentation used by the capacity 
building projects. An accompanying narrative d^^scribes the 
included materials. 
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f Final Design Report for the Evaluation of the State Capacity 
Bunding Grants Program (July 1977) is a two-volume report. 
Volume I contains the proposed designs for the full-scale 
evaluation. Volume II contains proposed instrumentation. 

• Building Capacity in Dissemination; Literature Review (March 
1977) was used to inform the design process. The Literature 
Review consists of tv/o separate but related producTs^ The 
first summarizes dissemination literature, using an organiz- 
ing framework which parallels that followed in NTS design 
work. The second product consists of an extensive biblio- 
graphy and outline of topics covered in the Review . Each 
entry in the outline is followed by a list of relevant cita- 
tions. 

By describing and evaluating the process of developing dissemination 
capacity in selected SEAs and by assessing the program's effects, the NTS 
study has provided basic information for the improvement of state dissemina- 
tion efforts, developed mechanisms for the continual evaluation and measure- 
ment of dissemination capacity, and by so doing, attempted to enhance the 
development of a nationwide dissemination system or configuration for improv- 
ing educational practice and enhancing educational equity. 
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1 

INTRODUCTION 



The purpose of this substudy is to provide documentation regarding the- 
development and interpretation of the scales which were designed to measure 
the various facets of state dissemination systems, NTS Research Corporation, 
under Contract No. 400-76-0166, Building Capacity for Improvement of Educa- 
tional Practice: An Evaluation of NIE's State Dissemination Grants Program, 
collected information on the activities of states as they developed dissemi- 
nation capacity. These data were subjected to a scaling analysis for for the 
following purposes: 

f To confirm that indicants grouped together on the basis of 
their content were, in fact, measuring the same thing. 

• To eliminate indicants not related to the other indicants in 
the facet. 

f To parsimoniously summarize a large body of data 

t To serve as a measuring device, by assigning facet scores to 
each scale for use in characterizing state capacity. 

The scales were first presented in the Interim Report of this contract 
(NTS, 1979). As we examined and interpreted the scales for that report, we 
noted a possible pattern of development of capacity in many of the scales 
which would indicate that a similar process was occur ing across states. If 
so, that developmental process would be of interest to Investigators of or- 
ganizational dtvelopment. It would be potentially useful to NIE as a tool 
for monitoring, for providing technical assistance, and to assist in making 
awards. We were unable to investigate these potential uses of the scales for 
the Interim Report. This substudy presents the results of NTS' further 
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refinement of the scales and assesses the potential utilization of the 

scales. The rest of this report is organized into four major sections: 

Section 2, Conceptual Framework , in which the hypothesized rela- 
tionships between various domains that may influence the build- 
ing and utilization of capacity are described, along with the 
placement of the scales in this framework j 

Section 3, Scaling Methodology , where the procedures used to 
develop the scales, including the selection of items for the 
scales and the assignment of scores for each state on the scales, 
are discussed; followed by an interpretation of each resultant 
scale. 

Section 4, Capacity Building as a Developmental Process , where 
the degree to which the scales provide a developmental perspec- 
tive to the capacity building process is assessed; and 

Section 5, Interpretation and Utilization , where the interpreta- 
tions that can be drawn as well as those which should not be 
drawn from the scales are discussed. How the interpretations 
might be utilized for purposes of program improvement is also 
discussed. 

The first half of Section 3^ Development of the Scales , is a relatively 
technical discussion of how the scales were developed and how Rasch scaling 
analysis was employed. For those readers less interested in this technical 
discussion we suggest that this half could be omitted without losing the 
essence of the substantive discussion. We do suggest, however, that the sec 
ond half of the section. Interpretation of the Scales , be reviewed for an un 
derstanding of the content of the scales. 

There are three companion volumes to this report: 

Building Capacity for Educational Improvement. An Evaluation of 
NIE State Dissemination Grants Program. 

1979 State Abstracts , a document which profiles dissemination 
activities in thirty-eight SEAs; and 

A Study of Linker Agent Activities and Roles . 
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CONCEPTUAL FRAMEWORK 



As a part of the approach to meet the basic objectives of this study, a 
cop.:eptual framework was developed which specified and organized the varia- 
bles which define the process of building dissemination capacity. The con- 
ceptual framework is presented in Figure 2.1. The framework includes five 
components and specifies the relationships between the components. The five 
components of the framework are: (1) State and SEA Contextual Characteris- 
tics; (2) NIE Program Characteristics; (3) State Capacity Building Project 
Characteristics; (4) Facets of an SEA Dissemination System; and (5) Outcomes: 
System Outputs and Impacts. These components can be further classified into 
three major categories: Context, Process, and Outcomes. 
The Context Domain 

Component 1: Contextual Characteristics . The first component refers to 
variables which describe contextual characteristics of the particular state 
and SEA. State characteristics include such variables as state size, exis- 
tence and use of intermediate service agencies or regions, school enrollment, 
number of school districts, and modernity (Herriott and Hodgkins, 1973) SEA 
characteristics include such variables as attitudes in the SEA towards 
change, centralization (Wirt, 1977), previous and current involvement of the 
SEA in dissemination activities, and the relative influence of the SEA, in- 
termediate education agencies (lEAs), and Lt'As in local educational improve- 
ment. 

Component 2: NIE Program Characteristics . The second component pre- 
sents factors which characterize the NIE Program, including its design and 
operations at the Federal level. Included are Program goals. Program 
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emphases, grant award criteria, project funding policies, and project monitor 
roles , 

The variables contained within Components 1 and 2 of the conceptual 
framework may be considered as the context domain, which includes both the 
legal/policy framework and the social/political setting within which the pro- 
jects are situated. The legal/policy framework includes not only the SCBP 
and guidelines associated with the Program, but also other Federal and state 
dissemination programs and policies. 
The Process Domain 

Component 3; State Capacity Building Project Characteristics . The 
third component contains variables which describe the resultant program in- 
tervention at the SEA level; that is, the project's structure and activities. 
Project structure includes such variables as the project's number of years in 
the SCBP, funding, location within the SEA, project director tenure, and man- 
agement arrangements. Project activities include such factors as targeting 
clients for dissemination activities, building additional information files, 
working with lEAs to provide linkage to local school districts, and interact- 
ing with NIE. 

Project characteristics may be considered as comprising the process do- 
rrain, and include project structures and activities as well as other SEA dis- 
semination activities. It must be noted that the actions taken by the state 
and the SEA are activities which tend to obscure the relationship between the 
project and the building of dissemination capacity. Phrased in another way, 
the project is provided "leverage" money through which a wide range of activ- 
ities are generated, activities which often extend beyond the project. At 
the same time, the state/SEA may also be developing additional activities 
outside of the project which enhance the dissemination system. 
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The Outcomes Domain 

The outcomes domain includes two categories of variables within the con- 
ceptual framework: 1) Facets of an SEA Dissemination System; and 2) Dissemi- 
nation System Outputs and Impacts. The outcomes domain includes not only 
those variables which are appropriate objectives for this study to measure, 
that is growth of dissemination capacity, but also those which reflect the 
long range goals of Federal policy as stated by NIE. 

Component 4; Facets of an SEA Dissemination System . This component 
refers to those elements which comprise a dissemination system: the re- 
sources, linkers, linkages, services, and institutional arrangements an SEA 
develops, implements, and institutionalizes to improve local educational 
practice and enhance educational equity. An SEA dissemination system, of 
which the capacity building project is usually just a portion, is depicted as • 
being comprised of six facets. Facet 1, Comprehensive Resource Base , refers 
to the types of resources (e.g., ERIC, promising practices files) that the 
SEA has the ability to access. Facet 2, Comprehensive Linkages , refers to 
the availability and use of a variety of individuals (e.g., SEA staff, lEA 
staff, LEA staff) and media (e.g., radio, television, publications) to con- 
nect educators with the information and services needed to improve local 
school practice. Facet 3, Coordinated Resource Base , refers to the extent to 
which various mechanisms for coordinating resources (e.g., a formal referral 
P'ucess, knowledge by resource base personnel of other resources, use of 
other resource bases) are available and used. Facet 4, Coordinated Linkage , 
refers to the extent to which linkers coordinate their activities with re- 
sources and linkages in order to provide assistance to improve local school 
practice. 
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^3cet 5, Institutionalization , refers to the extent to which the SEA 
dissemination system is developed so that it remains after NIE funding of the 
SEA'S capacity building project terminates; the provision of funds by the 
state for dissemination and statements by the chief state school officer 
supporting dissemination are examples of variables included in Institu - 
tionalization. Facet 6, Equal Access/Equal Op port unity , refers to the extent 
to which the SEA dissemination system provides resources to all educators on 
all topics, and the targeting of dissemination efforts to individuals who 
work with special populations, including the handicapped, minorities, and 
women. 

Component 5 : Dissemination System Outputs and Impacts , refers to out- 
puts of the SEA's dissemination system and their effects on the improvement 
of local practice and enhancement of educational equity. These systems out- 
puts and impacts relate to the longer range goals of improving local educa- 
tional practice and enhancing educational equity. An investigation of these 
outcomes was beyond the scope of the overall study; however, when possible, 
we presented descriptions in Volume 1, Final Report , indicative of the effect 
of dissemination capacity building upon these outcomes. 
Uses of the Scales 

Having described the relationship of the dissemination system scales to 
the conceptual framework, we suggest that the scales can serve an important 
purpose for this evaluation of the Program. The scales provide us with 
measures of a d-'ssemi nation system. Through the use of the scales we can 
conduct analyses which search for those factors that influence the extent to 
which the dissemination capacity is developed and the dissemination system 
implemented. We will describe in the following chapters the development of 
the scales and investigate their ability to assist us in assessing dissemi- 
nation capacity and implementation. 
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Development of the Scales 

This chapter describes the procedures used to develop a scale for each 
facet of the SEA dissemination system and to assign scale scores for each 
state. The purpose of each scale was to allow measurement of all of the 
states on a particular facet. However, for this to occur, the scores of the 
states on eac-h scale had to be comparable to one another. The ability to 
compare states was important, as one of the major purposes of the study was 
to assess what impact state and project characteristics had on making one 
state different from another with respect to the building of dissemination 
capacity. Thus, the indicants (i.e., individual items) that were selected 
for each facet had to be measuring the same thing (e.g., comprehensiveness of 
resource base) for all of the states. This meant that indicants that did not 
appear to be measuring the same thing had to be eliminated, even though they 
might reflect an important part of one or two states' dissemination systems. 
It was necessary to measure what the states had in coimon with respect to 
each facet, and to ignore, therefore, what might be a unique approach that 
some states were taking towards building capacity. The elimination of such 
indicants in no way represented a judgment as to the appropriateness of a 
state's approach. Rather, it simply meant that those indicants were not 
reflective of the facet that the rest of the indicants in the scale were 
measuring. It would have been inappropriate to combine an indicant that was 
measuring one thing with a group of indicants that were measuring something 
else in the computation of a facet score. 

Each scale was compribed of the set of indicants that best measured that 
facet. The indicants and states were then ordered on the scale. A state's 



placement on a scale reflected the degree to which a facet had been developed 
(i.e., the attainment of resources or activities) by that state. 
Procedures 

In order to produce a quantitative description of each state's dissemi- 
nation system with respect to these six facets, it was necessary to develop 
scales that would measure a state's position on each of the scales. These 
scales were based on the responses of SCBP directors to the Capacity Building 
Indicants questionnaire (CBI) that was administered in 1978 and 1979. The 
responses from the 1979 administrative were used to verify the analyses based 
on the 1978 data. Since all of the directors indicated that no restrictions 
existed with respect to Equal Access/Equal Opportunity, there was no need to 
develop a scale for this facet; all of the states would have fallen at the 
highest point on this scale. 

The scale for each facet was developed using a three-step process, with 
the second and third steps being repeated until a final set of indicants was 
identified that measured that facet and only that facet. In the first step, 
indicants were selected and grouped into scales on the basis of their con- 
tent. In the second step, the empirical interindicant relationships were 
used to eliminate those indicants in each scale that did not seem to be mea- 
suring the same dimension as the other indicants in the scale. In the third 
step, the scores of the states on the individual indicr/nts were submitted to 
a latent trait scaling analysis, and if necessary, additional indicants were 
el iminated. 

Step One; Content analysis . In the first step, a content analysis of 
the available indicants was conducted to determine the set of indicants which 
appeared to measure the same facet. The criteria for inclusion in this step 
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were fairly liberal, in order not to eliminate an indicant that might be 
strongly related to other indicants even when this was not iirmediately obvi- 
ous from the indicant's content. 

Step Two: Interindicant relational analysis . In the second step, the 
interindicant correlations were used in a reliability analysis to assess the 
degree to which the indicants in each scale were all measuring the same 
facet. The coefficient alpha (KRgo^ reliability was computed for each 
scale. Two measures were employed to assess the extent to which a particular 
indicant was measuring the same facet as the other indicants. First, corre- 
lations were computed between each indicant and the total of all the other 
indicants in the scale. Second, KR20 coefficients were computed exclud- 
ing each indicant, in turn. Indicants which had low indicant-total correla- 
tions or which lowered the reliability of the scale were excluded. 

Step Three: Scaling analysis . The indicants retained after the relia- 
bility analysis were submitted to a latent trait analysis using the Rasch 
(1960) scaling model. This model postulates that the probability of a state 
having or not having a particular indicant (i.e., a positive response or a 
negative response) is a function of the. difference between the state's score 
on the scale and the indicant's score on the scale. If the state's score is 
higher, the probability of a positive response will be greater than .5 (i.e., 
greater than a 50/50 chance), while if the state's score is lower, the proba- 
bility of a positive response will be less than .5 (i.e., less than a 50/50 
chance). The state's score on a scale reflects how much c» rdination, com- 
prehensiveness, or institutionalization its dissemination system has, and is 
based on the state's total number of positive responses with respect to the 
indicants of the scale. The indicant's score on the scale reflects how 
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difficult it is for any state to have a positive response on that indicant, 
and is based on the states' total number of negative responses with respect 
to that indicant. For example, the scale measuring the comprehensiveness of 
the resource base had a number of resources arranged on the scale, with the 
ordering of the resources directly related to how many states possessed each 
of those resources. 

For a state to be relatively high on the scale, therefore, it must have 
positive responses to more of the indicants than most of the other states. 
This is an indication that the state is higher on the facet than are most of 
the other states. For the indicants, this relationship is reversed. The 
more states that have negative responses on a particular indicant, the higher 
the scale value of that indicant will be. For an indicant to have a large 
number of positive responses implies that the indicant cannot be too high on 
the facet being measured because so many states possess that indicant. In- 
dicants with very few positive responses, on the other hand, will be highest 
on the facet, reflecting the fact that very few of the states possess a suf- 
ficiently high amount of the attribute in order to score positively on the 
indicant. 

The Rasch model analyzes all of the states' responses to all of the in- 
dicants and places the indicants and the states on the same scale. Since the 
Rasch analyses are based on positive/negative responses, all of the indicants 
were dichotomized, except for the Comprehensive Resource Base indicants, 
which were already dichotomized as provide/do not provide. The responses to 
the other indicants were based on a five point scale, with a "1" meaning 
"very limited,''' or "nonexistant" or "never," and a "5" meaning "very exten- 
sive" or "always." To dichotomize these indicants, we combined the "very 
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limited" and "liimted" categories into negative responses, and the "moder- 
ate," "extensive," and "very extensive" categories into positive responses. 

The dichotomized responses were then submitted to a Rasch scaling compu- 
ter program called BIGAL (Wright and Mead, 1979). In addition to computing 
scale values for both the states and the indicants, this program also pro- 
vides several measures of the degree to which the indicants are all measuring 
the same facet. The first of these statistics, the "total Ntest," measures 
the extent to which the predicted responses for that indicant matches the 
observed responses for that indicant. The higher the value, the lower the 
match. Wright and Mead suggest a value of 2.0 as the cutoff for this total 
_t-test statistic, beyond which, they say, the statistic is indicating a poor 
fit of the predicted responses for that indicant to the data. A second sta- 
tistic, the "between _t-test," is based on dividing the states into two 
groups, those with high scores and those with low scores, and assessing the 
degree to which the predicted average response to an indicant for each group 
matches the observed average response to the indicant for each group. As 
with the total t^-test statistic, the higher the value, the poorer the fit; 
again we followed Wright and Mead's suggestion that a value of 2.0 be used as 
a cutoff. All indicants with either a total jt-test statistic or a between 
t^-test statistic of greater than 2.0 were eliminated. 

Steps Two and Three were repeated for the remaining indicants. This was 
necessary because whenever some of the indicants were eliminated, the statis- 
tics for the remaining items were altered. Since the statistics all reflect 
the degree to which an indicant is measuring the same facet as the other in- 
dicants, they are sensitive to what indicants are placed in the "other" cate- 
gory. Consequently, an indicant might not have originally detracted from the 



scale's reliability or the scale's fit to the Rasch model within the complete 
set of indicants. However, that same indicant might detract from the relia- 
bility or the fit of the scale when that scale is based upon a subset of the 
original indicants. If all of the indicants again passed .^.11 of the various 
tests of fit, then the process stopped. If not, then the appropriate indi- 
cants were eliminated, and Steps Two and Three were repeated once again. For 
most of the scales, only two iterations were required, and in no cases were 
more than three iterations necessary. 

When the final Rasch scaling analysis was completed, we examined the 
relationship between the raw scores of the states on each scale and the state 
scale values as determined by the Rasch analysis. We noted that the rela- 
tionship between the total "raw" scores and the scale scores was almost 
linear; i.e., a change of 1 raw score point for each scale tended to corres- 
pond with a change of .2 on the Rasch scale for virtually the entire range of 
raw scores. Consequently, for purposes of interpretabil ity, we decided to 
use the actual raw scores rather than Rasch scale scores in subsequent analy- 
ses. 

The output of the Rasch scaling analysis is an ordering of the states 
and the indicants on the same scale. The ordering of the states is fairly 
straightforward to interpret: A state is higher or lower than another state, 
which in turn implies i:hat the state has more or less dissemination capacity 
(iit least with respect to that facet) than does the other state. We must 
emphasize, however^ that this does not in any way place a value judgment on 
the jjosition of the state on t! j scale, a point that will be discussed at 
length in Section 4. 

With respect to the ordering of the indicants, the scale may suqgest 
some sort of developmental paradigm. It must be emphasized, however, that 
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the existence of a developmental paradigm is only suggested by the scale, and 
not confirmed by it. What the scale tells us is that if a state has a posi- 
tive response with respect to a particular indicant, then it is likely to 
have positive responses with respect to all of the other indicants having 
lower scale values. It may well be that this means that for a state to have 
a certain resource on the Comprehensive Resource Base scale, for example, it 
was necessary for it to first acquire other resources preceding it on the 
scale. On the other hand, the scale may simply be hierarchical, reflecting 
the possibility that some resources are easier to acquire than others, and 
that the number of resources that a state has may simply be a matter of the 
size of its budget rather than its position in some sort of developmental 
process. An empirical assessment of the developmental versus hierarchical 
paradigm was undertaken, and is described in Section 4. 

The 1978 and 1979 indicant responses for the various scales are display- 
ed in Figures 3.1 to 3.6. The institutionalization scale is displayed in 
Figures 3.6a and 3.6b, The scale is unitary (i.e., it is one scale) but it 
IS divided in order to express important information about the institutional- 
ization process. The indicants in Figure 3.6a are related to what we believe 
occurs during the initial stages of capacity building and centers upon pro- 
ject activities, while the indicants in Figure 3.6b are related to agencywide 
activities that occur later in capacity building. The figures display how 
many of the 25 states responded positively to each indicant in 1978 and in 
1979. The total _t-test fit statistics for the indicants are shown next to 
the indicant descriptions. For the 1978 scales, all the _t-test values for 
the indicants fall well below the 2.0 cutoff criterion suggested by Wright 
and Mead; the same is true for virtually all of the indicants in the 1979 
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scales as well. For those indicants to which either all of the states or 
none of the states had positive responses, the Rasch model will fit perfectly 
and therefore the fit statistic is meaningless. For these indicants, the fit 
statistic has been replaced with "**". 

The reliabilities for each of the six scales were calculated for the 
1978 and 1979 data. The KRgo "internal consistency formula was used, and 

the reliabilities are presented at the bottom of Figures 3.1 to 3.6. While 
it is true that reliability values can range from 0 (''o internal consistency) 
to 1 (perfect internal consistency), the interpretation of reliability coef- 
ficients must be tempered by an awareness of the number of indicants in the 
scale. The more indicant that a scale has, the more reliable it will tend to 
be. The nature of the relationship between reliability and test length is 
stated in the Spearman-Brown prophecy formula: 

r' = mr 

1 + (m-l)r 

where r is the reliability of the original test, m is the multiple of the 
original test length by which the test has been lengthened or shortened, and 
r' is the reliability of a test m times as long as the original. This formu- 
la can be used to re-express the observed reliabilities of the original 
scales in terms of scales that have a certain uniform number of indicants. 
In order to provide a context for comparison, we chose 30 as the number of 
indicants, since this is a typical number of items that appear in a 
nationally standardized test or subtest (e.g., reading vocabulary, reading 
comprehension, mathematics computation, etc.). Tests or subtests of such 
length are generally considered to be sufficiently reliable if their 
KRgo internal consistency measure equals approximately .90.^*^ 
Therefore, we have 

^•^ See, for example, Technical Bulletin 1 for the 1977 edition of the 
California Achievement Test, where a substantial proportion of the sub- 
tests have about 30 indicants and a KR20 Reliabilities of about .90. 
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presented in Figures 3.1 to 3.6 the reliability for a 30 indicant scale based 
upon the observed reliabilities of each scale in 1978 and 1979. These 30 
indicant reliabilities all tended to be quite high, indicating that the in- 
ternal consistency of these scales was on a par with the internal consistency 
of commercially available, nationally standardized tests and subtests. It 
might be noted that it would have been possible to select the indicants in 
the scales such that these reliabilities would have been even higher. How- 
ever, our primary criterion for indicant selection was that the indicants 
total Jt-fit fall below 2.0. While these two approaches will tend to yield 
similar sets of indicants, they will not be in perfect agreement. The relia- 
bilities displayed in Figures 3.1 to 3.6, however, show that even though a 
Rasch scaling criterion was used for purposes of indicant selection, the 
scales also did quite well with respect to the reliability measure associated 
with more traditional tesc theory procedures. Thus, it is clear that for 
each scale, the selected indicants all tended to be measuring the same thing. 

We examined the relative positions of the 1978 indicants to the relative 
positions of the 1979 indicants. The absolute positions of the 1979 indi- 
cants tended to be somewhat higher, reflecting that the 25 Cohort I, II, and 
III states had been in the program one year longer. However, we still ex- 
pected the relative positions to be the same. Consequently, we computed a 
Spearman rank order correlation coefficient for each scale comparing the 1978 
indicant ranks (i.e., the relative position of an indicant with respect to 
the other indicants in the scale) with the 1979 indicant ranks. These corre- 
lations are displayed at the bottom of each figure. These correlations are 
all quite high, indicating that the ordering of the indicants along each 
scale had not chan::&d appreciably from 1978 to 1979. The fact that the 
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ordering remamed reasonably invariant is exiremely important, as it implies 
that the set of indicants constituting each scale were measuring the same 
thing at both points in time. Not only does this mean that the indicant 
structure was replicated, but that it is meaningful to talk about the differ- 
ences in the 1978 and 1979 scores as reflecting change, since the same facet 
was being measured in the same manner both times. 

Interpretation of the Scales 

Having now discussed the procedures used to develop the scales and the 
psychometric properties of the scales, we now consider what sort of interpre- 
tations can be attached to the scales. Such an interpretation must necessar- 
ily be based on an examination of the selected indicants and their positions 
on the scales, but it is also relevant to consider what indicants were not 
selected as well. Accordingly, we present all of the indicants associated 
with each scale in Tables 3.1 through 3.6. In each table, the indicants as- 
sociated with the primary scale are shown first, followed by secondary scales 
that include groupings of unselected indicants which tended to be associated 
with one another, as evidenced by their interindicant correlations. These 
are followed by a list of the remaining indicants that appeared to have lit- 
tle relationship with any of the other indicants. 

It should also be noted that our discussion reflects an assessment of 
the content of the scales as it relates to the developmental paradigm; as we 
discuss the ordering of the indicants on each scale, we are also discussing 
the ways in which the states might proceed in the development of their dis- 
semination systems. 
Comprehensive Resource Base 

The breakdown of the Comprehensive Resource Hase scale into the various 
indicant subgroups is shown in Table 3.1. The primary comprehensive resource 
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TABLE 3.1 

COMPREHENSIVE RESOURCE BASE 

PRIMARY SCALE 

1. ERIC * Education Resources Information Center 

2. ECER « Execeptional Child Education Resources 

3. NICSEM/NIMIS - National Information Data Organizer 

4. NTIS - National Te^.hnical Information Service 

5. Federal legislation file 

6. Promising practices file 

7. Listings/descriptions of federal and state funded innovative programs 

8. Listings/descriptions of operating local exemplary initiatives 

9. SEA human resource file 

10. lEA hunan resource file 

11. LEA hunan resource file 

12. State legislation file 

13. SEA products 

14. NDN products 

15. PREP Packages - Putting Research into Educational Practice 
SECONDARY SCALE #1 

1. AIR/ARM - Abstract of Instructional and Research Material in Vocational 
and Technical Education 

2. Dissertation Abstracts 

3. Lab and center products 

SECONDARY SCALE #2 

1. Education Index 

2. FIDO - Fugitive Information Data Organizer 

3. Files on user needs in the state 

SECONDARY SCALE #3 

1. PA - Psychological Abstracts 

2. Sociological Abstracts 

3. SSCI • So';ial Science Citation Index (also called Social Scisearch) 
SLCONDARY SCALE #4 

1. State«produced instructional materials 

2. Institutes of higher education-^produced instructional materials 

3. Locally-produced instructional materials 

4. lEA-produced instructional materials 

REMAINING INDICANTS 

1. PIPS • Project Information Packages 

2. Right«to^Read Packages 

3. NYT/IB - New York Times Information Bank 



Possible responses were: Available^ Not Available 
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base scale includes the various knowledge resources within the resource base 
that a project accesses in responding to clients' requests. These include: 

1. ERIC and other national information files; 

2. Validated programs, including Federal and state-funded in- 
novative programs and local exemplary programs; 

3. Promising practices/program files, which refer to non- 
validated educational methods and ideas; 

4. Human resource files which match client need with avail- 
able consultants in the SEA, lEA, LEA, or IHE, and 

5. Federal and state legislative files. 

Examination of Figure 3.1 shows that in 1978 all projects had access to 
four resources: Education Resource Information Center (ERIC), National In- 
formation Center for Special Education Materials/National Instructional 
Materials System (NICSEM/NIMIS), National Diffusion Network (NDN) products, 
and SEA products. With the exception of NICSEM/NIMIS, the other three are 
resources that SEAs generally have available before funding. These four re- 
sources form a base upon which a comprehensive set of resources is further 
developed. Other resources typically available are descriptions of Federal 
and state-funded innovative programs and such files as National Technical In- 
formation Service (NTIS) and Exceptional Child Education Resources (ECER). 
Resources that about fifteen to twenty of the projects have are: files of 
promising practices, including local exemplary programs and Putting Research 
into Educational Practice (PREP) packages; legislative files; and SEA human 
resources. The least frequently included resources (utilized by less than 
half of the states) are LEA- and lEA-generated local human resource files. 

The ordering of the indicants included in the comprehensive resource 
base shows two underlying patterns: One pattern involves the type of re- 
sources; the becond pattern involves the sources of the materials. The most 
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Indicants (fit*) 
1979 



Interwedlate Service Agency Human Resource 
File 



-0.74 



LEA Human Resource FHe -0.43 



Putting Research Into Educational Practice {PR£^>) 
PacKages 0.16 



SEA Huflvin Resource File -0.37' 

Federal Legislation File 



-0.37T_ 
-0.41 J 



Promising Practices File 0.30 

Information on Operating Local Initiatives Which Are 
ixemplary jjq 

State Legislation File 0.90 



Exieotlonal Child Education Abstracts -0 

Nit onal Technical Information Service (NTIS) . . -0 

Info. nation on Federal and State Funded Innovative 
ProgruW -0.34 

SEA Products "TfT" 

NONPrxJucts 

Educ .don Resources Info^^tion Cen' lEPlC) . . 
N^- lonal Instrvction Materials System: (NImis) . . 



.44 1 . . 

.06 J 



5-. 
6 - • 



.12- - 
--13-- 

- -15. . 

to 
20 



- -22 - ■ 



23 - • 
2i " . 



Intermediate Service Agency Human Resource 
File 



-0.41 



LEA Human Resource File 0.73 




SEA Human Resource File -1.18 

Putting Research into Educational ^rjctice (?R£P) 
Packages 0 

State Legislation File -0.73 



Promising Practices File 0.95 



Federal Legislation File Q,96 

Infornrtation on Operating Local Initiatives Ahic^ Are 
Exemplary ^0.18 

Information on Federal and State Funded Innovative 
Programs ' -0.75 



Exceptional Child Education Abstrscts -0.73 



SEA Products 0.4«5 

NCN VodiiCts C.Sl 

Education Resources Irfcr-iation Cente'- . . 

National Instruction Materials System 'SIVIS' . . 



Scale Reliability « .72 Scale Reliability = .52 

Scale Reliability for 30 Items « .84 Scale Reliability for 30 Items 

Spf .rman Rank Correlation - .90 

r^Jf^-^^^ t-test statistic - .hould be less than 2.0) 
ERJCirfect fit) 



.68 
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The most widely used resources are: national data flies, print-based materi- 
als, and products. Innovative programs (including Federal and state-funded 
innovative programs and local exemplary programs) are the next numerous. 
Promising practices files and legislative files are included with less fre- 
quency, while human resource files are the least often included resource in 
an SEA'S comprehensive resource base. With respect to source, national re- 
sources are most often a part of the dissemination system, followed by state 
resources, local district resources, and finally intermediate education agen- 
cy (lEA) resources. Notable for their absence are resources in institutions 
of higher education (IHEs). 

The first secondary comprehensive resource base scale includes AIR/ARM, 
lab and center products, and Hissertation Abstracts. The second secondary 
comprehensive resource base scale includes FIDO (Fugitive Information Data 
Organizer), lab products, and files of user needs. This configuration of re- 
sources tends to occur in systems that hav., a relatively small number of re- 
sources, suggesting that several states acquire this cluster of resources in 
conjunction with ERIC to form an alternative kind of resource base, rather 
than acquiring the kinds of resources found on tne primary scale. 

The third secondary comprehensive resource base scale includes Psycho- 
logical Abstracts, So:iological Abstracts, and Social Sciences Citation 
Index. All of these are bound abstracts and therefore do not represent re- 
sources that are distributed to clients but are instead utilized by resource 
base and project staff to locate relevant materials. 

The fourth secondary comprehensive resource base scale includes instruc- 
tional materials from various sources within the state: SEA, lEA, LEA, and 
IHE. 
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The remaining indicants include federally-prepared packages of which 
some are now obsolete, such as Project Information Package (PIP). 
Coordinated Resource Base 

The breakdown of the Coordinated Resource Base scale into the various 
indicant subgroups is shown in Table 3.2. The primary coordinated resource 
base scale describes a spectrum of behavior ranging from a broadening of 
awareness on the part of project staff of the various components of the re- 
source base in the SEA and LEAs to the development of referral and contact 
procedures to assure the availability of all extant resources for meeting 
cliant requests. Examination of Figure 3.2 shows that for both years, almost 
all of the states reported that the resource base and project staffs were 
aware of the components of the resource base, so that these appear to repre- 
sent the baseline for describing a coordinated resource base. As one reads 
up the scale, from those indicants which are used by more states to those 
used by fewer states, and if one assumes that those elements used in more 
statv^s are easier to develop than those used by fewer states, one can envis- 
ion the process of coordinating resources. Thus the following scenario, 
based on the ordering of the indicants and supported by our site visits, is 
suggested. 

At the earlier stages of coordination, the services to clients tend to 
overlao and the various resource bases duplicate each other 'a efforts. Oper- 
ationally, a client could access a variety of sources for the same informa- 
tion. This does not necessarily mean, however, that the centralized resource 
base had as comprehensive an information file in the area of special educa- 
tion, for instance, as the special education unit had. 

As the SEA and the other resource agencies become awaru of the services 
the project can provide and cooperation between the project and other 
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TABLE 3.2 
COORDINATED RESOURCE BASE 

PRIMARY SCALE 

1. Resource base staff awareness of components of comprehensive resource base 

2. Project staff awareness of components of a comprehensive resource base is 

3. A formal referral process that incorporates procedures to avoid duplication of 
effort 

4. Responses coupled with one or more referrals are 

5. The rate of rejections to responses are 

6 The number of contacts/referrals of requests to other source agencies are 

7. Usage of the compendium (documented compilations) of resources by other resource 
agencies is 

REMAINING INDICANTS 

1. SEA inner circle of administrators awareness of a comprehensive resource base is 

2. Other resource agencies (IHE's, museums, state sv^tem) awareness of project staff 
and services is 

3. Resource base service to Title IVC, special education, vocational education is 

4. Other resource agencies ( IHEs, 'museums, state library system) usage of project 
staff or services is 

5. Project awareness of other sources contacted by client is 

6. Redundancy in types of information services available from different resource 
agencies is 

7. Communications between linkage agents and resource base(s) are 

8. Multiple source response to single source (e.g., ERIC) responses are 

9. The frequency of coupling a human resource file and research information files 
in responding to client requests is 

10. Agencywide understanding of a common definition of dissemination is 



Possible responses were: Non Existent, Limited, Moderate, Extensive, Very Extensive 



O i 

3-16 

ERIC 



Figure 3.2 
COORDINATED RESOURCE BASE 



Indicants (fit*) 
1978 

Rati of fttjectlons to ^tspon$«$ 



Usage of the Compendium of Resources by Other 
Resource Agencies .1^86 

Format Referral Process Th&t Incorporates 
Procedures to Avoid Duplication of Effort . 0.42 

Frequency of Coupling Human Resource File 
and Research Information Files in Re- 
sponding to Client Requests -0.70 

Number of Contacts/Referrals of Requests 

to Other Resource Agencies -o.62 ' 

ResDonses Coupled with One or .'^ore 

Referrals -2.i0 ' 



Redunaancy in Types of InfomTation Services 
Availaole from Different Resourct> Agencies 1.41 



?''o:ec: Sta'f Awareness of Ccmocnents of 

Ccnorenensive Resource Base 

Resource Base Sta^^ Awareness of Cormoiients 

0^ Comorehenslve Resource Base ^* 



'No. of States 



9 

:o 
n 



:3 - • 



1 



Indicants (fit*) 
1979 

Rate of J^ejections to Resoonses ** 



usage of the Compendium of Resources by Other 
Resource Agencies -0.27 

Redundancy In Types of Information Services 
Available from Different Resource Agencies -0.25 

Formal Referral Process That Incorporates 
Procedures to Avoid Ouplication of Effort . 0.71 

Number of Contacts/Referrals of Requests to 

Other Resource Agencies ..... -2.46 

Frequency of Coupling Human Resource File 
and Research Information Ffles in Re- 
Lsoonding to Client Requests -o.20 

J?esponses Coupled witn One or Vcre 
Referrals 



-0.20 



!5esource Sase Staf -bareness c** :orDonents 

of CofTorenens 1 ve Resource 3ase ..... O.lc 



'^oject Sta^* Hwarffiess :^ Ccmconents 
Coniorenens i ve ^esourca Sase 



scale Reliability » .61 Scale Reliability = .46 

Scale Reliability for 30 Itans = .85 scale Reliability for 30 Items 

Spearman Rank Correlation = .75 



.76 



eric! **'''°*^^ t-test statistic - should be less than 2.0) 
I **Perfect fit) 
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resources is achieved, the central resource base expands its services and 
broadens the variety of materials which can be included in the response to a 
client's request. This is accomplished by coupling responses with referrals 
to other agencies which may have more extensive sources of information per- 
taining to the client's request. The development of coordination/cooperation 
is further achieved when the central resource base begins to contact these 
other agencies for the client rather than referring the client to the other 
agency. At this point the response sent from the project to the client in- 
cludes information from a variety of sources. 

This "reaching out" of the project to other agencies or program units 
develops working relationships which ff^rm for the SEA a "compendium of re- 
sources," a network of units which begin to share information. This sharing, 
as shown in our site visits, can take the form of cooperative agreements be- 
tween program units to respond to these requests for information or it can 
take the form of other program units providing the information sources to the 
central resource base for inclusion directly into centralized files. 

As greater coordination is achieved, other resource agencies become 
aware of the project and its services and begin to utilize the project to 
collect information for their clients. Finally, a formal referral process is 
developed through which the SEA can respond with a minimum of duplication of 
effort. In practical terms, when there is adequate coordination of the SEA 
resource base, a client can request assistance from any program unit in the 
SEA network and receive a comprehensive (i.e., data from multiple sources) 
response. 

The remaining indicants form a hodgepodge. They include indicants that 
were rejected before scaling procedures because they either, (1) unnecessar- 
ily duplicated items in other scales (i.e., "agenc)^ide understanding of a 
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common definition of dissemination" was also included in institutionaliza- 
tion); (2) were vague and subject to differing interpretations (i.e., "multi- 
ple source response for single source responses"); or (3) did not pertain to 
all the states (i.e., "coupling a human resource file with a research infor- 
mation file" would not indicate coordination between resource bases if the 
human resource file was also included in the resource file or if the state 
did not have a human resource file). Other indicants probably did not corre- 
late with the primary scale because they focused on the awareness and usage 
by the other resource agencies of the project's resource base, whereas the 
primary scale's focus is on the project's actions in coordinating and using 
these agencies. 
Comprehensive Program Linkage 

The breakdown of the Comprehensive Program Linkage scale into the vari- 
ous indicant subgroups is shown-, in Table 3.3. The primary comprehensive pro- 
gram linkage scale includes three groups of linkages that appeared to reflect 
relationships which were interorganizational in character: 

1. dissemination specialists, including resource base staff, 
NDN staff, and Title IV staff; 

2. program-specific specialists, including the staffs of such 
programs as: special education, career education. Title I, 
handicapped education and early childhood education; and 

3. state library system staff. 

Examination of Figure 3.3 shows that for both years, the most frequently de- 
veloped linkages are resource base staff. Title IV-C, and NDN staff, who are 
often referred to as "dissemination-type specialists." The least developed 
linkage elements are with the staffs of the state library system and migrant 
education and early childhood education programs. 
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TABLE 3.3 

COMPREHENSIVE PROGRAM LINKAGE ELEMENTS 

PRIMARY SCALE 

1 . Resource base staff 

2. TUle I staff 

3. Special education staff 

4. NON staff 

5. Tltie IVC Gtaff 

6. Early childhood education staff 

7. Career education staff 

8. Migrant education staff 

9. Handicapped education staff 
10, State library system staff 

SECONDARY SCALE #1 

1. LEA representatives 

2. Building level representatives 

3. Local school librarians 

4. School board members 

REMAINING INDICANTS 

1. State associations (content^onented) staff 

2. Other state agency staff 

3. Regional education center staff 

4. Intermediate service agency staff 

5. Vocational education staff 

6. AduU education staff 

7. Content-oriented specialists (SEA) 

8. Urban education staff 

9. Right-to-Read staff 



Possible responses were: Unavailable, Not Used, Limited Use, Moderate Use, 

Extensive Use, Very Extensive Use 
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Figure 3,3 
COMPREHENSIVE PROGRAM LINKAGE ELEMENTS 



Indicants (fit*) 
1978 



•is* of ^igranc Education Sta^^ 0. 



Ob 



use of £ar!y Childhood £d.jcat1on Staff . . . -0.96 

'Jse of State Library System Staff 0 '2 

Use of Hanrjicaoped Education staff 2 la 

use of rule I Staff ; ; .o!o6 



Use of Career Education Staff . 
Jse of Special Education Sta*' 



-1.63 1 
-0.73 J" 



vse of NCN Sta'^ 

JSC of Title !VC Staff 0.C3 

Vse of Resource Base Staff 0. 7 



'»o, Of' i talcs 



• • 3 

m * ^ 

• M 

•n 

-'5 - • 
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Indicants (fit*) 
1979 



use of .^igv-ant Education Staff .o.91 

Jse of State Library System Staf^ Q.65 

•Jse 0^ larly Childhood Education Staff .... 0*81 

'^se 'it!* : Scaf-' 

.se 3f W 3:af' p^^^ 

.se of Career Education Sta'f o.40 

'Jse of 3:ecM' Ziuci^^on Sti^* -C.23 

-se of Handicjpoed Education 5:a-' . . , . , -l.i: 



kit 



Scale Reliability « .79 crA}p PoH;iKi-i .-f v. 

scale Reliability for 30 Ite^s - ..92 Sc^l' R^ll^b];!?]; ^o^ 30 Ite^s • 88 

Spearman Rank Correlation = .81 

( *Total t-test statistic - should be less than 2.0) 
'tr^erfect fit) 



Several trends can be identified between the 1978 and 1979 scales. One 
trend that seems clear is that projects generally first involve elements that 
are close to it (i.e.» at the SEA) and then proceed to enlist persons at 
levels closer to local education, most notably intermediate education agency 
staff. While the involvement of NDN staff experiences a slight decline, the 
involvement of Title IV staff and many program-specific staff generally in- 
creases. Possibly as a result of further implementation of Public Law 94-142 
(Education for Handicapped Children Act), special education and handicapped 
education staff showed the most dramatic increase in involvement, followed by 
Title I staff. 

The secondary comprehensive program linkage scale includes linkages at 
the local levels, including LEA and building level representatives, local 
librarians, and school board members. While the primary scale reflects in- 
terorganizational linkages, the. secondary scale represents local linkers who 
serve linker functions. 

The remaining indicants include: (1) content-oriented specialists, in- 
cluding SEA staff and members of content-specific state associations; (2) re- 
gional staff; and (3) staff of such programs as vocational education, adult 
education, and urban education. It may be that content-oriented specialists 
and particular program staff are more difficult to establish relationships 
with due to "turf" battles. Regional staff is a poor indicant since it is 
redundant with lEA level staff represented in the primary scale. 
Comprehensive Media Linkage 

The breakdown of the Comprehensive Media Linkage scale into the various 
indicant subgroups is shown in Table 3.4. The primary comprehensive media 
linkage scale includes print-based materials and electronic devices that are 
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TABLE 3.4 

COMPREHENSIVE MEDIA LINKAGE ELEMENTS 

PRIMARY SCALE 

1. Project-specific publications 

2. SEA publ ications 

3. Newspapers 

4. Educational television 

5. Audiovisual aids 

6. Computer-based user systems 

7. Slides 

8. Films 

9. Prerecorded cassettes 
SECONDARY SCALE #1 

1. Closed circuit television 

2. Commercial television 

3. Radio 



Possible responses were: Unavailable, Not Used, Limited Use, Moderate Use, 

Extensive Use, Very Extensive Use 
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Figure 3.4 
COMPREHENSIVE MEDIA LINKAGE ELEMENTS 



Indicants (fit*) 
1978 



..se of Films -0.60 

.iSe of Educational Television -0.60 

use of Prerecorded Cassettes -I.IO - 

wse of Sewspaoe'^s -0.01 • 

jse of Comoucer-3ased User Systems O./l • 

■Jse cf Slides -1.37"! 

vse of Audiovisual Mas -0.37J 



use of Project Soecific ?ubl1cattons . . , . 0.59 
■jse of SEA Publications 0.14 



No. of Stares 



M 
12 
13 
■l-U 

17 

- :3 



-•21 



* -24 



Indicants (fit*) 
1979 



use of rnms 0.02 

use of Prerecorded Cassettes -0.39 

Use of Educational Te'evis^gn -0.51 

use of Newspapers 'C.29 

Jse of Computer-Sased Jser Systems .... 1.31 

use of Slides -2.60 

use of Audiovisual Aids -1.31 



Use 0*"' Project S?ec:fic ^.jjj; .^at ^ops 



jse 0^ SEA PuDlications 0.04 



Scale Reliability « .80 Scale Reliubility « .63 

Scale Reliability for 30 Items « .93 Scale Reliability for 30 Items « .84 

Spearman Rank Correlation « .96 



cD?r"Total t-test statistic - should be less than 2.0) . 
L^^PprfprT fit) ail 



utilized by projects to create awareness and interest in the project's ser- 
vices and as a vehicle for delivering information. Print-based materials in- 
clude SEA and project publications and newspapers. Electronic media include 
slides, audiovisual aids, cassettes, educational television, and films. 

Examination of Figure 3.4 shows that, for both 1978 and 1979, the pre- 
dominant types of media are project and SEA publications. Each of the other 
media types are used by less than half the projects. Projects, in general, 
have about four or five of the nine types included in the scale. It appears 
that after the use of publications, projects will explore various other media 
forms, and eventually select two or three media types that suit their needs 
the most, rather than attempt to acquire a wide repertoire of available media 
1 inkages. 

The secondary comprehensive media linkage scale indicants include closed 
circuit and comnercial televisi&n and radio, all of which are generally com- 
mercially-produced, rather than SEA-produced^ media sources. 
Coordinated Linkage 

The breakdown of the Coordinated Linkage scale into the various indicant 
subgroups is shown in Table 3.5. The primary coordinated linkage scale de- 
scribes the coordination of the linkers utiMzed by the project with the re- 
source base and with the inter-organizational linkages. The ordering of the 
indicants illustrates a broadening of awareness and usage of linkage networks 
by linkers and project staff, as well as the development and implementation 
of linker training. One of the most challenging tasks facing the management 
of an SCBP project is the integration and coordination of the personal linker 
agents and resources in order to bring information to the client. The order- 
ing of the indicants in Figure 3.5 shows that this coordination process 



TABLE 3.5 
COORDINATED LINKAGE ELEMENTS 

riMARY SCALE 

1. The proportion of personal linkage agents who are aware of components of the linkages 
used by the project i 

2. The proportion of personal linkage agents who are aware of linkage services available 
through the project is 

• 3. The proportion of project staff who are aware of components of the linkage system is 

4. The proportion of all personal linkage agents who use the compendium of resources is 

5. The usage of the compendium of resources by all personal linkage agents is 

6. In responding to client requests, the coupling of personal linkage agents and other 
resources is 

7. The usage of the compendium of linkages by all personal linkage agents is 

8. The usage of the compendium of linkages by project staff is 

9. Communication flows between linkage agents and the resource base(s) are 

10. The usage of training programs for personal linkage agents is 

11. The development of training programs for personal linkage agents is 

SECONDARY SCALE #1 

1. The proportion of administrators within the top three levels of the SEA who are aware 
of the components of the linkage system is 

2. The long range planning for coordination of satallite linkages with linkages services 
of central facilities is 

3. The amount that LEA committees determine priorities and information needs of local 
educators is 

4. Recognition of typical problems (e.g., overload, marginality, delivery problems, 
monitoring problems) is 

5. The times a client must contact many sources in order to have his/her information 
needs met is 

6. The amount that duplication of effort is avoided due to a formal referral process is 

7. The addressing of typical problems (e.g., overload, marginality, delivery problems, 
monitoring problems) is 



Possible responses were: None or Very Low, Low, Moderate, High, Very High 



4^7 



3«26 



o 

ERIC 



Indicants (fU*) 
1979 



usage of the Comoendium Linkages by All 

Personal Linkage Agents -1.10 

Usage of the Compendium of Resources by 

All Personal Linkage Agents *1.62 

Proportion of All Personal Linkage Agents 

uho Use the Compendium of Resources 0.54, 

•Jse of Training Programs for Personal 

Lif^Kage Agents 1.56 

;sage of tnc Compendium of Linkages by SC3 

Project Staff -0.76 

Proportion of Personal Linkage Agents 'nl\Q 

Are Aware of Components of the Linkage 

System jsed by SC3P . ' -0.77 , 

Devetopmont of Training Programs for Personal 

LinKage Agents '*0.39 

Conmunication Flows 3etween Linkage Agents 

and Resource 8ase(s) 0.73 

Coupling of Personal Linkage Agents and 

Other Resources in Responding to Requests . . 0.70 , 

Proportion of Persona) Linkage Agents Who 

Are Aware of LinKdge Services Available 

Through :ne SC3P -0.78 

Proportion of SC8 Project Staff **^ho ^re Aware 

or Components of LinKage System 0.18 
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12 
13 
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'5 
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21 
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Proportion of All Personal Linkage Agents ^ho 

use the Compendium of Resources -l.ei 

Jsage of the Compendium of Linkages cy All 

Personal Linkage Agents .... I -2.29 

Development of Training Programs for 

Personal Linkage Agents 2. II 

Usage of the Compendium of i^ssources by Ail 
Perirenal LinKage Agents -G.27 

' UScige of tne Compendium of uinkages by SCS 

Project Scaf^ -0.40 

Jse 0^ Training "rcgrams for Personal LinKage 
Agents 3.46 

Proportion or Personal ,«inKage Agents nho 

Are Aware of Comoonents of tne Linkage 

System used by SC3P C.04 

'Proportion of Personal Linkage Agents who Are 
Aware 0 Linkage Services Ayallaole "hrougn 

the iCS? C.02 

Comnunic.ulon f^lows Between Linkage Agents 

and Resource Baseis* *. . . . 2. HI 

ProDO'-tion 3''" SCE ^''o ec: Sta*** .%nc -re Aware 

of Conoonencs of .inK.ige SysteT 

Couc'ing "ersonal linkage Acents 3*^0 C*^e** 
, Resources in ^esconol'^c :o =ecues:s ** 



Scale Reliability « ,87 Scale Reliability = ,78 

Scale Reliability for 30 Items « ^95 Scale Reliability for 30 Items « ,89 

Spearman Rank Correlation « .77 

EWC*Total t-test statistic - should be less than 2.0) 



requires the following steps: (1) project staff must become aware and knowl- 
edgeable of potential resources and linkage components that might be helpful 
to the project; (2) project staff must then seek out these components and be- 
gin to utilize them (i.e., "test them out"); (3) linkers must be trained, 
with training which ranges from awareness of the available linkages and re- 
sources to how to utilize them and how to effect school improvement; and fin- 
ally, (4) linkers must familiarize themselves with the myriad of resources 
and linkages, and then proceed to use those that "fit" client needs. 

The coordination of linkage activities refers to the extent to which the 
three major structural components of the system (linker, linkages, resource 
base) are coordinated in order to provide the most efficient and effective 
method of allowing the linker to access the system and to bring information 
to the client. It includes awareness and usage of available linkage networks 
by project staff and by linkers and the training of linkers, where the train- 
ing emphasis corresponds t) the functions that linkers are expected to ful- 
fill. Table 3.5 also illustrates a broadening of awareness and usage and the 
increased coordination between the resource base and the linkers. Client re- 
quests are responded to through the interaction of resource base personnel 
and linkers. 

While the major coordinated linkage scale includes four si ages within 
the process of coordinating linkers and resources, two aspects that seem to 
bracket this process are included in the secondary scale. At one end is 
awareness and planning for linker services; at .ne other end is the recogni- 
t'i">n and addressing of typical problems encountered by linkers and duplica- 
tion of effort. 
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Institutional Izatlon 

The breakdown of the Institutionalization scale into the various indi- 
cant subgroups is shown in Table 3.6. The primary institutionalization scale 
describes the process by which the project's functions and activities are 
continued after the grant period. The scale includes two major parts. One 
portion is project-specific, and describes various stages of initial activi- 
ties in the adoption of the project within the SEA, ranging from awareness, 
interest, evaluation, trial, and adoption (see Rogers, 1971). The second 
part describes the gaining of support within and then outside the SEA to in- 
sure its permanent status, including agencywide planning for dissemination, 
on-paper commitments, and increased funding provisions. Examination of the 
ordering of the indicants in Figure 3.6a and 3.6b suggests that the scale may 
be viewed as encompassing two separate but related parts. The first part 
(3.6a), as represented by the elements in the lower range of the scale, is 
project-specific and parallels Rogers' stages of adoption. This portion of 
the scale describes initial activities which should lead to institutionaliza- 
tion, such as developing efforts to create awareness and interest in the pro- 
ject and generating a demand for services. The second part (3.6b) of the 
scale, as represented by the top half of the scale, describes the project's 
attempts to gain support within and then outside the SEA, namely attempts to 
insure institutionalization. Probable hallmarks of acceptance of the project 
include: the gaining of representation in executive team sessions, under- 
standing by personnel both within and outside the SEA of the project's func- 
tions, and provisions made for gradual increases of support during the grant 
period and after the grant period ends. 

The second portion of the scale describes the later phases of institu- 
tionalization of the dissemination function beyond the identity of the 



3-" 50 



TABLE 3.6 

INSTITUTIONALIZATION 



PRIMARY SCALE 

1. Planning on an agencywide basis for dissemination is 

2. Understanding of a comnon definition of dissemination on an agencywide basis is 

3. The provision for state funding of project activities after the grant period ends is 

4. Efforts by the project to stimulate increased demands are 

5. Efforts to gain support from clients, potential support groups, and others within the 
larger organization are 

6. The project's "conversational credibility** In the SEA (e.g., project functioning is 
recognized and interest is shown in it) is 

7. The centralization of management of dissemination activities is 

8. The provision for gradual Increases of state support throughout the project period is 

9. Documentation regarding the project (e.g., position papers, role descriptions, stand' 
ard operation procedures, quality control procedures, rationale) is 

10. Budgeting on an agencywide basis for dissemination is 

11. State board action on dissemination other than action related to project funding is 

12. The consideration of the dissemination function in regular planning activities in the 
SEA is 

13. The mentioning of dissemination in the SEA goals is 

14. The contribution of other federal and state funding sources to a coordinated function 
of dissemination is 

15. The amount of state legislation dealing specifically with dissemination is 

16. The awareness by those outside the SEA of the functions being performed by the pro- 
ject is 

17. Efforts to create awareness among clients, potential support groups, and others with- 
in the larger organization are 

18« The amount dissemination is mentioned in state board goa.s is 

19. The understanding by those In the SEA of the role behaviors performed by those in the 
project is 

20. Mentioning of the function of dissemination in the state superintendent's annual 
report is 

21. The running of articles related to projeci activities in agency publications is 

22. The extent to which the project was planned by an agencywide group was 

23. The participation in executive team sessions, oi^ their equivalent by someone who is 
closely related to the project and designated as official dissemination representa- 
tives is 

24. The amount that the project's five year plan is updated as needed and regularly made 
available is 

25. The function of dissemination is located on the organization chart of the SEA 

26. There is an agency line item specific to the function of dissemination in the SEA 

REMAINING INDICANTS 

1. The involvement of project staff in preparation of dissemination Dlans for those fed- 
eral and state plans requiring dissemination (e.g., 94-142, IV-C, NON) is 

2. The constancy of '•titled** roles within the SEA related to dissemination despite per- 
sonnel changes is 

3. The involvement of the project director in responding to the NON solicitation was 

4. The training and recruitment for well defined role positions related to dissemination 
in the SEA is 

5. The acceptance by those outside the SEA of the functions being performed by the pro- 
ject is 



Possible responses were: Non Existf^nt, Limited, Moderate, Extensive, Very Extensive 
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Figure 3.6a 

KEEPING THE SYSTEM GOING (INSTITUTIONALIZATION, PART 2) 



Indicants (fit*) 
1978 



Awtjnt of State legislation Oeallng Specifically with 
Dissemination 0,73 



Agency Line Iteffl Specific to the Function of 
Dissemination in the SEA . . 0.43* 

State Board Action on Dissemination Other Than Action 

Related to SCB Project funding .0.19 

Budgeting on an Agency wide Basis for 

Dissenrt nation j2 

Mentioning of the Function of Oissemination in the 

State Superintendent's Annual ^^eoort 0.75 

Amount Oissftfllnatlon is Mentioned In State Board 
Goals 0.82 

Understanding of a Common Definition of Dissemination 
on an Agencywide Basis 0.97 

Contribution of Other Federal and State Funding Sources 
to a Coordinated Function of Disseminacion . . . -1.90 

Mentioning of Dissemination in SEA Goals 0.28' 

Consideration of the Dissemination Function in Regular 

Planning Activities in the SEA -1.90 

Centralization of Management of Dissemination 
Activities 0.4A 



Planning on an Agencywide 9asls for 
Dissemination 



.17 



Function af Oisseminacion is Located on tne 
Organliation Chart of the SEA 0.49 



'^0, of States 



Indicants (fit*) 
1979 



^ount of State Legislation Dealing Soeclfically with 
Dissemination o.33 




Mentioning of the Function of Dissemination In the 

State Superintenoenc's Annual ??eport • 0«10 

Agency Line Item Soeclfic to the Function of 
Dissemination in the SEA o*87 

State Board Action on Dissemination Other Than Action 

delated to SC8 Project Funding o*00 

Amount Dissemination 1$ Mentioned in State Board 
Soals h06 

Planning on an Agencywide Basis for 

Dissemination .1,72 

Unoerstandlng of a Common Definition of Dissemination 

on an Agencywide Basis 

Centralization of Management of Dissemination 

Activities •0.28 

Sudneting on an Agencywide Basis for 
. Dissemination -0.08 

Contribution of ^ther Federal and State Funding Sources 
J Coordinated Function of Disseminattcn . . . 0.73 

^'ertloning of Dissemi nat'fcn <n SEA Goa^s .... -LgO 
Function Dissemination is Located or the 
. Organization Chart o^ the SEA .Q.9"» 



.ii^vrc*^ of the ^*sse«'i'^2t'cn 'j«c:<c?^ "i ^e^u'ar 
-l5rni'*g ActWities in tie Zl^ *o.J 



f"]® S^Jl^bility . .89 Scale Reliability = .82 

Scale Reliability for 30 Items » .90 Scale Reliability for 30 Items = 86 

Spearman Rank Correlation s .85 



ER|Cta1 t-test statistic - should be less than 2.0) 



Figure 3.6b 

GETTING THE SYSTEM GOING (INSTITUTIONALIZATION, PART 1) 



Indicants (fit*) 
1978 



No. Of States 



Indicants (fit*) 
1979 



■ 11 



srovlslon for State Funding of SC8 Activities After 
the jrant Period Ends '^'^^ 

iwar«ness by Those Outside SEA of Functions Being 
Perfoni«d by SC3 Project "'^^ 

Anount That Project's =1ve Ytar Plan Is Updated as 
Needed and Made Available Regularly -O-o^ 

Extent to ymch SC8 Project was Planned by an ^ 
Kr;jf SSle; ^eiai^ to'sb'p;ojeit'Ait;v;. 
ties in Agenr.y Publications u.^c j 

oarMcioatlon In Executive Team Sessions, op Their 
Equivalent ay Someone who is Closely Relates o 
Project and Designated as Official Oissemination^^^ 
Representative . . • 

Provision for Gradual Increases of State Support 
Throughout the Project t>er1od •^•'^^ 

m 

•Ondtrstarding 5y Those In SEA of «ole Behaviors 

Performed ^>y Those in SCB Project "O-i^ 

Oocjmentation Regarding Project 'J'i^- 

SCS Project ''Conversational Credibility' m 

SEA 

Efforts to Gain Support from Clients. Potential 
Support Groups, and Others WUhin the Larger^ ^^^^^^ 

EffSrirbJ^SCs'proJect St'inn!ilate increas*»d 
OeJ^nds • * * ■ 

Efforts to Create Awareness Among Clients. Potential 
Support Groups, and .Others Within tne Larger 
Organization 



.12 



•13 




•22 



Provision for S!:ate Funding of SCB Activities After 

the Grant Period Ends • • • J^*^^ 

Provision for Gradual Increases of State S ipport 

Throughout the Project Period -O^l* 

Extent to Which SC8 Project was Planned by an 

.Agencywide Group •Q*^^ 

Punning of Articles Related to SCB Project 

ties in Agency Publications -O.wU 



AriKJunt -ha: Voject's ^^/e ^ear ^lan is Updated as 
Needed and ?^ade Available Regularly -u.u^ 

^arMcioation in Executive Team Sessions, or Their 
Eauivalent oy Someone who Is Closely Related to 
Project and Designated as Official 01 ssenii nation 
Representative ^'^^ 



r Awareness by Those Outside SEA of Punct^ons Seine 
5*rforned oy SCB Project • • • ! ' "'^^ 

jnSerstandi'g by Those in SEA of ^ole 3ehavlor< 

^ oerforrned by ^hose io SCB Project 



- Efforts by SCS Project ".o Stimulate Ircrease^^ ^^^^ 

Demands / \* * \ 1*21 

_ Oocjmentation Regarding Project ^•^^ 



r ^^^^orts ^0 Gal'^ SuDDort frcm CHe'^ts. ^otert'.al 
SuDOCrt Grcuos. and Others ^i^Mn t'^e '.arier 



>c3nl:at*on . • ' . « ; ' 

SCS ^rcect "Ccnversafc^a^ Creci?!:iVty . 
-ffor^s'to Create Aware'^tss Att^cng C^e'^ts. ^oten^a^ 
Sucrort Vouos. anO Ot-er»s .^It*'"^ -a-ce*- 
^ Organization 



^^-.ale Reliability ^ .89 

t.toef statistic - should be less than 2.0) 



Scale Reliability * .82 



project. Three groups of indicants describe this process: planning for dis- 
semination on an agencywide basis, on-paper commitments to a general dissemi- 
nation function, and increased funding provisions. The most systematic pro- 
cess involves initial goal statements in the agency, then planning activities 
that capitalize on project input (in terms of role definitions and experi- 
ences), mechanisms for coordinating funding for dissemination, and increasing 
conmiitments for future funding. Planning is expanded from planning for the 
project and its activities to planning for dissemination on an agencywide 
basis. This may involve a temporary or permanent centralization of manage- 
ment and representation of the project in executive team sessions for plan- 
ning. There is usually an increase of "on-paper" commitments to a general 
dissemination function, including goal statements (by the SEA, CSSO, and/or 
state board), state legislation, location of the dissemination function on 
the organizational chart, and an agency line item specific to the function of 
dissemination. Increase in funding occurs as the state specifically budgets 
for dissemination, utilizes funds from either Federal, state, or other 
sources, and makes provisions for support of project activities both uuring 
and after the grant period. 

Thus, the institutionalization scale describes two phases of the insti- 
tutionalization process: adoption of the project by the SEA, and institu- 
tionalization of the dissemination function by the agency. Implementation, 
that is, how the project builds dissemination capacity and what that capacity 
looks like are captured by the previously described scales. 

The remaining indicants include two items that assessed the involvement 
of project staff in other dissemination efforts and t'„u that involved the 
extent to which dissemination roles are well-defined. One possibility for 
their exclusion from the scale is that they are not necessary activities 
leading towards institutionalization. 



4 

CAPACITY BUILDING AS A DEVELOPMENTAL PROCESS 

In Section 3, we discussed the content of the scales in some detail. In 
those discussions we noLed that th^re appeared to be a pattern to the activi- 
ties expressed in the scales. These patterns, such as the early acquisition 
of materials most easily obtainable to finally obtaining materials that are 
more difficult to develop, seemed to indicate a sequence of behaviors to 
which we suggested a developmental paradigm. However, our interpretation of 
the content of the scales was not sufficient for verifying a developmental 
sequence. Therefore one of the purposes of this section is to examine this 
developmental assumption. 

In the Interim Report , we presented scales based upon data collected at 
one point in time (i.e., data were from a cross-sectional field collection 
effort). This enabled us to describe projects and dissemination systems, but 
not how they grow or develop over time. However, because data were collected 
from projects and SEAs that had been in the State Capacity Building Program 
for different numbers of years, we considered the possibility that these pro- 
jects were in different phases in developing their dissemination capacity. 
In fac':, moderate correlations had been found between the number of years in 
the Program and most of the scale scores. Therefore, we considered the pos- 
sib-ility that the scales could be used, to describe a developmental sequence 
that states go through in the course of building capacity. One of our major 
objectives is, of course, to trace the development of systems over time. If 
we can verify that the scales reflect the developmental patterns or stages 
within various aspects of a system, then we have a general developmental pat- 
tern and a means for assessing individual or group differences. Wo can do 
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this by comparing scale scores for states and qroups of states on the basis 
of cohort, time of measurement, and years in the program, and on contextual 
factors including initial dissemination capacity, size, and so forth. 

While a developmental interpretation is relatively easy to apply to the 
content of the scales, such an interpretation is also fraught with serious 
proDlems. Consider, for example, the Comprehensive Resource Base Scale. It 
was intuitively appealing to consider the development of the resource base as 
proceeding through the acquisition of nationally-available data bases (e.g., 
ERIC), moving through the stages of collecting materials which are more dif- 
ficult to track down and obtain (e.g., local exemplary program files) to the 
development of materials which the project may have to develop on its own 
(e.g., a human resources file). However, there are real problems in making 
such conclusions on the basis of these cross-sectional data, since develop- 
ment might not follow that pattern over time. In order to more accurately 
assess the actual pattern of development, projects needed to be examined 
across time periods. We needed to verify whether the ordering of the indi- 
cants represents a developmental progression in the building of capacity 
within particular facets, or whether the scales are a cumulative frequency 
that measures the quantity and rarity of each indicant. For example, the 
higher up an indicant is, the rarer it might be, rather than representing a 
characteristic that is added to a system at a later point in a project's 
life. Interestingly, while the scales measuring the comprehensiveness of 
resources and of program linkages are more demonstrably valid, they are also 
more likely to be interpreted as measures of rarity. The scales measuring 
coordination and institutionalization seem more likely to show progression in 
a developmental sequence. 

r - 1 ,-^ 
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In order for the scales to be measuring a developmental process, a 
state's position on the scale at one point in time should be predictable from 
a knowledge of that state's position on the scale at a previous point in 
time. A necessary condition for a scale to be developmental is that the 
scale be hierarchical. A scale is hierarchical if a positive score on a par- 
ticular indicant 1s accompanied by a high probability of positive scores on 
all of the other indicants below it on the scale. Conversely, a negative 
score on a particular indicant is accompanied by a high probability of nega- 
tive scores on all of the other Indicants above it on the scale. As a con- 
sequence, the indicants on a hierarchical scale are ordered by their degree 
of difficulty. In order to utilize the scales to measure growth. It 
is necessary to verify whether the scales are just hierarchical and measure 
the quantity and varity of each indicant's occurrence, or whether the order- 
ing of the indicants Is not only a hierarchical but a developmental progres- 
sion in the building of capacity within a particular facet. If the former 
case Is true, then this means that the higher an indicant is on the scale, 
the rarer It is, as opposed tc the indicant representing a characteristic 
that is added to a facet at a later time In the building of capacity. 

As a further clarification of the distinction between hierarchical and 
developmental scales, consider two measurements that we frequently make of 



As introduced in Section 3, the difficulty of an indicant is direct- 
ly related to how many states had negative responses with respect to that 
Indicant. The more states that respond negatively to an indicant, the 
more difficult it presumably is for any given state to respond positively 
to an indicant. What is meant by a positive response will of course vary 
from scale to scale. For the Comprehensive Resource Base Scale, a posi- 
tive response indicates that ci particular resource is available through 
the dissemination system. For' Institutionalization, a positive response 
signifies that an activity associated with institutionalization occurs 
frequently or extensively in the state. In summary, an indicant with a 
higher degree of difficulty is associated with fewer positive responses; 
an indicant with a lesser degree of difficulty is associated with more 
positive responses. 
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ourselves; body temperature and height. The f«ver thermometer scale is 
hierarchical; when the mercury is at 98.6, it is also at every degree marking 
below 98.6; by the same token, if the mercury is not at 98.7, then it is not 
at any degree marking above 98.7. The fever thermometer scale, however, is 
certainly not developmental, as an individual's temperature is a function not 
of the individual's age but rather of various life events, such as being ex- 
posed to an influenza virus. Our measuring tape scale, on the other hand, is 
not only hierarchical but is also developmental. The height scale is hierar- 
chical because if one is taller than 6 feet, one is also taller than 5 feet 
11 inches, 5 feet 10 inches, etc. However, it also measures a developmental 
sequence because from the time we are born at least generally up through our 
teenage years, the process of maturing includes growing taller. 

In Section 3, we established that the indicants within a scale form an 
order which is consistent not only across states, but also across two points 
in time. Since this order was consistent, we can therefore say that the in- 
dicants for each scale form a hierarchy, and that hierarchy is invariant 
across time. This means that the scales are appropriate not only as devices 
with which the states can be measured and ordered with respect to how much of 
each dissemination system facet they possess, but also with which the growth 
of the states can be measured with respect to each of the facets. In the re- 
mainder of this section, we consider whether the indicants for each scale 
constitute a developmental sequence. 

In order to assess whether the scales reflect a developmental paradigm, 
the scale scores should show a consistent relationship with age. "Age" in 
this context refers to the number of years that a state has been in the 
capacity building program. Consequently, it is necessary to have measures of 
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the states at at least two points in time in order to perform this assess- 
ment. When measurement is available from only one point in time, we can only 
perform a cross-sectional analysis. Such an analysis allows us to test 
whether there are differences in scale scores for states from different co- 
horts (i.e., of different project ages) measured at the same time, but such 
differences will be completely confounded with any cohort differences that 
may exist. Such cohort differences will be systematic in nature (e.g., the 
states in one cohort may have had substantially greater initial dissemination 
capacity than the states in another cohort), but will be completely undetect- 
able in a cross-sectional design. A second factor that will also be com- 
pletely confounded with growth will be the point in time at which the devel- 
opment of the system is measured. For example, if our measures were taken in 
a year following a period in which the dissemination community became aware 
of the importance of building a comprehensive? resource base, then all of the 
states might have immediately acquired as many resources as possible. The 
sudden emphasis on acquiring resources might have comple^ely altered any 
potential differences in number of resources between the states as a factor 
of the number of years that each state had been in the capacity building 
program. 

In order to investigate the extent to v/hich the scales reflect a devel- 
opmental paradigm (i.e., a relationship between program effects and number of 
years in the program) as opposed to the possible confounding of effects of 
cohort and/or time of measurement, a paradigm suggested by Schaie (1965, 
1973) is particularly useful. Schaie's paradigm permits the investigation of 
the development of systems by considering the degree to which that develop- 
ment is affected by three components: years in the program, the point at 
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which capacity is measured, and the cohort to which a state belongs. Table 
4.1 graphically depicts the paradigm, including cohorts, project ages, and 
measurement points, and will serve as a heuristic device in understanding the 
discussion which follows. 



Table 4.1 
Analytic Paradigm 



Cohort 


Ages 


I 

II 
III 
IV 
V 

Non-SCBP 


0 1 \J ^'^^'^-^^ 

0 r^-^-^^-^.^ 

n r 

0 
0 


Time of 
Measurement 


1975 1976 1977 1978 1979 

---Cohort-Sequential Method Data 
---Time Sequential Method Data 



The first of these procedures which Schaie refers to as the cohort- 
sequential method involves examining the data enclosed by the dotted line in 
Table 4.1. This data configuration requires information on projects drawn 
from two cohorts that have been in the program for either two or three years. 
Using this data, we can assess the effect of years in the program, and see if 
this effect remains constant across two different cohorts. In order to 
assess the average age change over one year starting at age two, as sampled 
in cohorts I and II, we use the formula: 

AVERAGE AGE DIFFERENCE = 

[(Age 3 Coh I) - (Age 2 Coh II)] -f [(Age 3 Coh 1 1 ) - (Age 2 Coh III)] 

2 

To assess whether this effect remains constant over two different cohorts, we 

use the formula. 

COHORT DIFFERENCE = 

[(Age 3 Coh I) - (Age 2 Coh I)] - [(Age 3 Coh il) - (Age 2 Coh II)] 
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If we should find that the effect of age is different for Cohort I than 1t is 
for Cohort II (i.e., the cohort difference is not equal to zero), then the 
usefulness of the results will be questionable. This approach, however, 
requires us to assume that the time of measurement has no effect. If this 
assumption is not met, then the differences that we might ascribe to years in 
the program could instead be attributed to time of measurement effects. If, 
for example, a bre.^kthrough occurred between 1977 and 1978 and another break- 
through occurred between 1978 and 1979, then the growth that we might see 
would be attributable to exposure to these breakthroughs, and not to the ef- 
fect of the program. 

The second of these procedures which Schaie refers to as the time- 
sequential method involves examining the data enclosed by the solid line in 
Table 4.1. In this data configuratitri, we have information on projects that 
have been in the program for two years or three years drawn at two points in 
time. Consequently, we can assess the effect of years in the program and see 
if this effect remains constant across two different points in time. In 
order to assess the net age change over one year starting at age 2, as sam- 
pled at Measurement Times 1978 and 1979, we use the formula: 

AVERAGE ACiE DIFFERENCE = 

[(Age 3 Coh I) - (Age 2 Coh 11)1 + [(Age 3 Coh II) - (Age 2 Coh III)] 

2 

To assess whether this effect remains constant over two different points in 
time, we use the formula: 
TIME DIFFERENCE = 

[(Age 3 Coh I) - (Age 2 Coh II)] - [(Age 3 Coh II) - (Age 2 Coh III)] 
If we should find that the effect of age is different in 1978 from what it is 
in 1979 (i.e., the time difference is not equal to zero), then the usefulness 
of the results will be questionable. This approach, however, requires us to 
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assume that the cohort membership has no effect. If this assumption is not 
met. then the differences that «e might ascribe to years in the program could 
instead be attributed to cohort effects. 

Each method, if used alone, confounds growth with either cohort member- 
ship or time of measurement. Utilizing both methods to assess the effects of 
growth allows us to "zero in" on the effect of years in the program. Specif- 
ically, if the differences tnat we observe in both methods are in fact due to 
increasing years in the progr™. and not to cohort membership or t,me of mea- 
surement, then we should observe exactly the same relationship between the 
scale scores and number of years in the progr^ regardless of which method is 
used. To the extent that confounding does exist, either due to cohort mem- 
bership or time of measurement, then combining these two methods should allow 
us to assess the strength of this confounding and to temper our conclusions 
about the effects of years in the program accordingly. 

unfortunately, we do not have the complete information needed to imple- 
ment the cohort-sequential method. Information was collected only in 1978 
and 1979, so the information needed for the cohort-sequential method from 
1977 is unavailable to us. Consequently, our estimate of average age differ- 
ence has to be based only on one pair of scores rather than two. For exam- 
ple, the effect of age change over one year starting at age two can be esti- 
mated only by [(Age 3 Coh II) - (Ag. 2 Coh II)]. Furthermore, we are unable 
to directly compare the effects of age across two different cohorts to insure 
that the effects of age are not confounded with cohort membership. To the 
extent, however, that our estimates of the age effect tend to converge, then 
TIME DIFFERENCE, from the time sequential method, will be zero, and we can 
assume that the COHORT DIFFERENCE, from the cohort sequential model, would 
also probably have been zero. 
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From our data, we can estimate the effects of net age change over one 
year for projects starting either at age one or at age two. The information 
that we will use is described in Tables 4.2a through 4.2f. 



Table 4.2a 
Comprehensive Resource Base 



Cohort 


Age 


Mean 


SD 


n_ 


Aqe 


Mean 


SD 




I 


3 


12.8 


.83 


7 


4 


11.28 


1.38 


7 


II 


2 


11.7 


2.55 


13 


3 


11.8 


1.70 


13 


III 


1 


11.0 


2.76 


5 


2 


11.0 


2.83 


5 


IV 


0 








1 


8.3 


2.08 


3 


V 


0 








0 


6.8 


2.28 


5 


Non-SCBP 


0 








0 


9.0 


2.0 


2 


Time of 
Measurement 


1978 


1979 



Table 4.2b 
Coordinated Resource Base 



Cohort 


Age 


Mean 


SD 




Age 


Mean 


SD 




I 


3 


4.86 


1.73 


7 


4 


5.0 


1.51 


7 


II 


2 


5.61 


1.55 


13 


3 


5.61 


1.59 


13 


III 


1 


4.6 


1.85 


5 


2 


5.6 


1.50 


5 


IV 


0 








1 


5.0 


1.0 


3 


V 


0 








0 


2.8 


1.30 


5 


Non-SCBP 


0 








0 


2.5 


1.5 


9 


Time of 
Measurement 


1978 


1979 



Table 4.2c 
Comprehensive Program Linkage 



Cohort 


Aqe 


Mean 


SD 


in 


Age 


Mean 


SD 


n 


I 


3 


7.71 


1.48 


7 


4 


7.28 


1.66 


7 


II 


2 


5.31 


2.94 


13 


3 


6.69 


2.27 


13 


III 


1 


6.4 


2.58 


5 


2 


7.2 


2.64 


5 


IV 


0 




mm ^ 




1 


3.0 


3.61 


3 


V 


0 








0 


3.0 


2.82 


5 


Non-SCBP 


0 








0 


8.50 


.5 


2 


Time of 
Measurement 


1978 


1979 
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Table 4. 2d 
Comprehensive Media Linkage 



Cohort 


Age 


Mean 


SD 


n_ 


Age 


Mean 


SD 


£ 


I 


3 


4.0 


2.0 


7 


4 


4.71 


2.18 


7 


II 


2 


3.38 


2.27 


13 


3 


3.54 


1.86 


13 


III 


1 


3.0 


3.52 


5 


2 


4.0 


1.78 


5 


IV 


0 






m 


1 


2.0 


2.0 


3 


V 


0 








0 


3.0 


3.32 


5 


Non-SCBP 


0 








0 






2 


Time of 
Measurement 


1978 


1979 



Table 4.2e 
Coordinated Linkage 



Cohort 


Aqe 


Mean 


SD 




Age 


Mean 


SD 




I 


3 


7.86 


3.83 


7 


4 


9.42 


1.18 


7 


II 


2 


7.77 


2.75 


13 


3 


8.46 


2.50 


13 


III 


1 


7.6 


3.38 


5 


2 


10.2 


1.6 


5 


IV 


0 








1 


5.0 


5.0 


3 


V 


0 








0 


3.4 


3.51 


5 


Non-SCBP 


0 








0 






2 


Time of 
Measurement 


1978 


1979 



Table 4.2f 
Institutionalization 



Cohort 


Age 


Mean 


SD 


£ 


Aqe 


Mean 


SD 


n_ 


I 


3 


17.7 


8.68 


7 


4 


18.43 


5.92 


7 


II 


2 


15.5 


4.46 


13 


3 


16.07 


3.85 


13 


III 


1 


13.6 


2.92 


5 


2 


20.0 


2.83 


5 


IV 


0 








1 


13.3 


9.5 


3 


V 


0 








0 


6.2 


3.70 


5 


Non-SCBP 


0 








0 


3. J 


2.5 


2 


Time of 
Measurement 


1978 


1979 
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It might be noted that if information had been collected from the Cohort 
IV states in 1978, then it would have also been possible to estimate the ef- 
fect of one year's growth for cohorts beginning at age zero. Without the 
1978 Cohort IV information, the possible effects of growth between ages zero 
and one are completely confounded with cohort effects and time of measurement 
effects. 

Also included in Tables 4.2a through 4.2f are the number of projects in 
each cell. Of the 29 projects in Cohorts I through III, 25 supplied informa- 
tion on the capacity building indicants questionnaire for both 1978 and 1979. 
Three of the four cohort IV states and five of the ten Cohort V states sup- 
plied information in 1979. Finally, we have information from two states that 
are not currently in the SCBP program. 

The impact of growth on scale scores is presented graphically and in- 
cludes all six groups (Cohorts I through V and Non-SCBP) for 1978 and 1979. 
Figures 4.1 through 4.6 each present the relationship between scale scores 
and number of years in the program. The mean scale scores for each cohort 
and time of measurement cell are presented as a function of years in the pro- 
gram. The mean scale scores from 1978 are represented by squares and joined 
together with line segments. The 1979 mean scale scores are represented by 
circles, and are also joined together by line segments for Cohorts I through 
V, while the non-SCBP average score has been left as distinct point. 

Given the small sample size and the resultant loss of statistical power, 
we decided that it would be more appropriate to examine the graphs in Figures 
4.1 through 4.6 in order to assess the developmental paradigm than to perform 
a series of statistical tests. In examining Figures 4.1 through 4.6, several 
considerations should be kept in mind. First, if the position on the scale 
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is solely a function of age (i.e., not a function of cohort or time of meas- 
urement differences), then each pair of points for years one, two, and three 
should lie extremely close together (e.g., the 1978 mean scores for cohort II 
should be close to the 1979 mean score for cohort III). To the extent that 
the line segments are superimposed on one another, the developmental hypothe- 
sis is supported and the cohort-sequential method and time-sequential method 
have converged. If the lines are not superimposed, then there are several 
other patterns that we should look for in an effort to understand the factors 
that might influence position on the scale. 

If the lines are more or less parallel to one another, then there are no 
time difference effects; i.e., whether the measurement was made in 1978 or 
1979 has no impact on position on the scale. Position on the scale, there- 
fore, is a function of some combination of age and cohort. Another pattern 
to look for when the lines are not superimposed is whether shifting the 1978 
points one year to the right will cause any of the points to be superimposed. 
Should a point be superimposed, the positioning of the projects on the scale 
is almost completely a function of cohort differences. This is because the 
two overlapping points will represent the same group of projects with one 
point being the 1978 average score and the other point being the 1979 average 
score. If no growth occurred between 1978 and 1979 for the same group of 
projects, then it seems likely that differences between the project averages 
in 1978 or in 1979 were attributable to differences between cohorts. 

The average scores for the various groups on the comprehensive resource 
base scale are presented in Figure 4.1. The first impression that this fig- 
ure conveys is that comprehensiveness of the resource base is a constantly 
increasing function of age, at least up through three years. Whether the 
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Figure 4.1 Comprehensive Resource Base 
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dropoff from three years to four years is truly a function of age or is 
unique to Cohort I projects cannot be tested from this information. Since 
the 1978 and 1979 lines are not superimposed, it would appear that age cannot 
be considered the sole determinant of scale position. Upon further examina- 
tion, In fact, it may well be the case that age is of little importance at 
least in the period of one to three years. Moving the 1978 line one year to 
the right superimposes not only the two Cohort II points, but also the two 
Cohort II I. points. In other words, from 1978 to 1979, neither Cohort II nor 
Cohort I changed on the comprehensive resource base scale. Thus, based on 
this information, the most likely explanation is that the differences between 
the project groups are attributable not to age differences but primarily to 
cohort differences. 

The average scores for the various groups with respect to the coordi- 
nated resource base scale are presented in Figure 4.2. As with the compre- 
hensive resource base scale, the first impression that this figure provides 
is that coordination is also a monotonically increasing function of age up 
through three years, with a dropoff from the third to the fourth year. Fur- 
ther examination indicates, however, that two different processes seem to be 
at work. The position of projects at one and two years of age on this scale 
would seem to be entirely a function of age. All three estimates for the 
effect of growing from one year to two years seem to have converged on ap- 
proximately one unit (approximately 4.8 to 5.6). This is clear support for 
the developmental metaphor. On the other hand, for both Cohorts I and II, 
there was virtually no growth from 1978 to 1979. It would appear that beyond 
two years of age, projects remain at the same position on the scale. The 
logical inconsistency in this argument, though, is that both of the Cohort I 
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points should therefore be at approximately 5.6, since that was the point 
that both Cohorts II and III reached at two years of age. More information 
than is currently available, especially the position of the Cohort I projects 
on this scaK orior to 1978, is needed to resolve this inconsistency. 

The group average scores for the comprehensive program linkage elements 
scale are presented in Figure 4.3. The validity of the developmental meta- 
phor for this scale immediately appears to be challenged by the configuration 
points. For both 1978 and 1979, Cohort I projects have the highest average 
scale score, followed by Cohort III projects followed by Cohort II projects. 
If we move the 1978 line one year to the right, the Cohort I points are ex- 
tremely close, while the 1978 Cohorts II and III points both lie approximate- 
ly t;ie same distance below the corresponding 1979 points. This would seem to 
suggest that there is a mixture of cohort and age effects operating here. 
Clearly, the Cohort I states did not show any growth. For the Cohort II and 
III states, the fact that the Cohort II states scored lower than the Cohort 
III states also suggests the presence of a cohort effect; however, both the 
Cohort II and Cohort III states each showed roughly the same amount of growth 
from 1978 and 1979 (approximately .5), thus suggesting that while cohort dif- 
ferences may have had an effect with respect to initial position, an addi- 
tional year in the program did have an impact on increased scale scores, at 
least for year 1 to year 2 and for year 2 to year 3. This is only a tenta- 
tive hypothesis, however, and would require at least another year's worth of 
information for its validity to be supported. 

The group average scores for the comprehensive media linkage scale are 
displayed in Figure 4.4. With the exception of the 1979 Cohort II average 
score, the impression that this figure presents is of a constantly increasing 
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Figure 4.4 Comprehensive Media Linkage 
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relationship between average scale score and number of years in the program. 
It should be noted in Table 4. 2d that each of the group average scores has a 
substantial standard deviation (close to or in excess of 2) associated with 
it. Given that the difference between the pairs of points at years 2 and 3 
is less than .5, it is difficult to determine whether the lack of overlap 
between the 1978 and 1979 lines is due to the confounding effec's of cohort 
and/or time of measurement, or is simply a function of measurement error. 
Regardless, the fact that the standard deviations are so large relative to 
the range of the scale renders any interpretation of the comprehensive media 
linkage elements data virtually meaningless. 

The average group scores for the coordinated linkage elements scale are 
presented in Figure 4.5. The 1978 line shows practically no differentiation 
between cohorts, while the 1979 line shows substantial but erratic growth. 
Given such a pattern, it is impossible to even guess at the relative contri- 
butions of age, cohort, and time of measurement without additional informa- 
tion. 

The group average scores for the institutionalization scale are present- 
ed in Figure 4.6. In general, there would appear to be substantial cohort 
effects, although there are some indications of an age effect in the growth 
from one to two years. If the 1978 line is moved over one year to the right, 
the Cohort I points are extremely close given the size of the standard devia- 
tions in Table 4.2f, as are the Cohort II points. In other words, there was 
little, if any, growth with respect to institutionalization for Cohorts I and 
II from 1978 to 1979. The substantial differences between the 1978 and 1979 
lines with respect to growth between one and two years further suggests the 
presence of a cohort effect. 
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Figure 4.5 Coordinated Linkage 
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It might be noted that the Cohort V group averages are consistently 
below the averages of all of the other groups. Whether this represents the 
distinction between being in the project for zero years as opposed to one 
year or more cannot be separated from the possibility that the Cohort V 
states, being the last to enter the program, were states which had not en- 
gaged in many dissemination efforts and therefore had a lower initial status 
than all of the other states. With the exception of the comprehensive pro- 
gram linkage elements scale, the Cohort V and Non-SC6P averages tended to be 
close together, indicating that the differences between states that have just 
entered the program and unfunded states are minimal. 
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CONCLUSIONS 



While the previous chapters of this substuay have focused on such de- 
tails as how the facets were conceptualized, how the facets were turned into 
scales, and how the developmental nature of the facets was assessed, the pur- 
pose of this chapter is to "step uack" from such detail and assess the over- 
all utility of these scales. Such an assessment of utility must necessarily 
be based on the extent to which iriformation regarding the scale scores of the 
dissemination system can be used for purposes of program planning and im- 
provement. 

To begin with, we believe that the development of the scales represents 
a significant methodological advance in understanding and conceptualizing the 
organizational attributes of a dissemination system. Typically, the organi- 
zation of a dissemination system has been described in qualitative rather 
than quantitative terms. A case study approach has frequently been used to 
generate a great deal of descriptive information regarding the system, which, 
when combined with information from other systems, could be combed for points 
of similarity and dissimilarity. Such an approach, while rich in detail, 
nonetheless makes it difficult for the individual to develop some sort of 
unified framework into which all of the systems can be placed. Such a frame- 
work permits .1 much clearer perception of system similarities and 
differences, and provide a much more concise description of each system. 
With such a description in hand, one can then begin to assess the impact of 
the environment and the implementation of the SCBP program on the dissemina- 
tion system's characteristics, as well as to assess the degree to which the 
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dissemination system characteristics result in effective utilization of the 
system, which in turn leads to better and better instruction. 

The legitimacy of the scales as instruments through which dissemination 
system attributes can be medbUred was established in Chapter 3. This 
legitimacy is based upon the observation that the ordering of the indicants 
within each scale was consistent across the states, thus demonstrating that 
all of the indicants within a scale seem to be measuring the same thing. 
Furthermore, the ordering of the indicants remained generally invariant 
across two administrations (fall of 1978 and fall of 1979) of the data 
collection instrument. The indicants within each scale, formed a hierarchy, 
where a positive response with respect to any indicant was accompaniv^d by a 
high probability that all of the other indicants lying lower on the scale 
would also be associated with positive responses. By the same token, a 
negative response with respect to an indicant was accompanied by a high 
probability that all of the indicants lying higher on the scale would be 
associated with negative responses. 

The descriptive power of such a set of scales is enormous. Rather than 
having to characterize a dissemination system through the atomistic approach 
of listing a number of details regarding that system, it is possible instead 
to adopt a more holistic approach, synthesizing all of the detailed informa- 
tion into a few scale scores. Knowledge of where a dissemination system lies 
on a scale automatically carries with it knowlpdge regarding the dissemina- 
tion system with respect to a number of details. If we regard the indicants 
as details, then the scale score infers which attributes the system is likely 
to possess, as well as which attributes the system is not likely to possess. 
In short, this substudy has revealed the presence of order among what might 
otherwise be regarded as a descriptive cacaphony of dissemination system 
attributes. 
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In addition to the succinct synthesis of a great deal of detailed infor- 
mation, the fact that the structure of the scales remains invariant both 
across states and acr.'js time has two important implications for how the 
scales can be utilized. First, states can all be ordered with respect to 
their score on each scale; in other words, the scores that two states receive 
on a scale can legitimately be used for purposes o" comparison, and conclu- 
sions can be drawn about one state being higher or lower or about the same as 
another state. As will be discussed shortly, some care needs to be exercised 
in drawing such conclusions, but, nonetheless, it is possible to use the 
scales to draw comparisons between two states or, for that matter, compari- 
sons between a state and the average scores of a group of states. 

Clearly, such comparisons need to be tempered by considerations such as 
how long the state(s) has been in the SCjP program, the demographics of the 
state, the size of the state, and the state's educational structure, to name 
but a few. What is important to note, however, is that the scales permit 
comparisons with respect to a few, salient attributes; without the scales, 
th'-j points of comparison would be either so numerous or so ill -defined as to 
render such comparisons virtually useless. 

In addition to being able to make comparisons between states or groups 
of states, it is also possible to use the scales to make comparisons between 
the positions of a state or group of states dt two or more points in time. 
In other words, the scales can be used to chart growth, because not orily are 
the scales invariant across states, but it has also been shown that the 
scales are invariant across time, at least for the years 1978 and 1979. 
Consequently, the same attribute of the dissemination system will be measured 
by one of the scales in multiple administrations of the instrument. Thus, 
the changes that a dissemination system goes through can be described in 
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terms of a few salient characteristics, rather than with respect to a multi- 
tude of details. As with the comparisons between states, utilization of the 
scales permits a concise description of the evolution of a state's dissemina- 
tion system. 

Having described the scales' potential for making comparisons between 
states and/or between points in time, three important caveats regarding the 
utilization and interpretation of the scales must be specified: first, no 
evidence currently exists that would justify making value judgments such as 
"good" or "bad" based upon whether a state's position on a scale is "high" or 
"low"; second, since the ccales were developed to be measurement instruments 
that could be applied to all states, they may be insensitive to dissemination 
system characteristics that are unique to only a few states; und, third, only 
occasional evidence has been found to support the interpretation of the 
scales as being not only hierarchical but also developmental. Wt discuss 
each of these caveats more fully. 

The first caveat concerns the "high/low" relationship with a jud&.iient of 
"good/bad"; it must be emphasized that such a relationship cannot be drawn 
from the data available to this study. Specifically, while the information 
regarding state dissemination systems with respect to a number of attributes 
was sufficient to construct a set of scales, value judgments regarding posi- 
tions on these scales must be based on some external criterion. Such a cri- 
terion might ultimately be the degree to which new knowledge is reflected in 
the improved quality of education in a state. Other criteria might be the 
extent to which the educators in the state are aware of new knowledge or, for 
that matter, are aware of eind utilize the dissemination systems. In the 
absence of any external criteria, it is unwise to even begin to speculate how 
a state's position with respect to the scales should be judged as "good" or 
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"bad." Given considerations such as geography, size of the school system, 
and political constraints, a combination of a moderately comprehensive re- 
source base and a moderately comprehensive linkage system might be most 
appropriate for one state, while for another state, little emphasis on 
resource base comprehensiveness with a heavy emphasis on linkage might be 
most appropriate. Furthermore, depending upon the degree to which a dissemi- 
nation system is developed, it may be that the position of the system with 
respect to certain scales is emphasized less, while the position of the 
system with respect to other scales is emphasized more. For example, during 
the initial stages of system development, rapid growth with respect to com- 
prehensiveness of the resource base might be particularly important, while in 
a later stage, the size of the resource base might actually shrink as only 
those resources deemed most essential are retained and monies are reallocated 
to develop other aspects of the system. 

In the absence of any external criteria, a value judgment about a 
state's position on the scale must necessarily be based on the observer's 
perceptions as to what constitutes an effective dissemination system. If 
such an observer is prepared to make such statements as, "the more resources 
you have, the better" or, "the more coordination you have, the better," then 
clearly, a high/low position on a scale can be translated into a good/bad 
judgment. We believe that such an approach to making judgments would be 
terribly simplistic, however, given the extreme diversity in needs and con- 
text that we have encountered in the course of this research. Given the 
influence that needs and context can have with respect to what sort of dis- 
semination system configuration will be maximally useful in improving 
instruction, a clear understanding of how these influences operate and the 
extent to which they are present in a particular state must be taken into 
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account before any value judgment regarding that state's dissemination system 
configuration can be rendered. 

The second caveat concerns the presence of unique dissemination system 
characteristics. Since the scales were developed to be applicable across all 
states, the scales could not include those attributes whose presence or 
absence in a state was not predictable from the presence or absence of other 
attributes in that state. While such a lack of predictability may be attrib- 
utable to the more or less random occurence of that attribute in any particu- 
lar state, it may also be attributable to the fact that, for a few states, 
possession of that attribute was particularly important, regardless of what 
other attributes they might or might not have. For example, a few states 
obtained the Fugitive Information Data Organizer (FIDO) as part of their 
resource base, an act which appeared to be completely independent of whether 
those states had a great many or only a few other resources. Consequently, 
FIDO was excluded from the comprehensive resource base scale. Thus, for 
those states subscribing to FIDO, the resource base scale does not completely 
reflect all of the resources that these states have at their command. Conse- 
quently, it must be recognized that while the scales are extremely powerful 
for purposes of comparison across states and/or across points in time, none- 
theless, the sensitivity of the scales to all of the attributes that can con- 
stitute a dissemination system is limited. In using the scales to make a 
comparison between two states, while in one sense the comparison is valid 
since the scales are measured in the same attributes in both states, in 
another sense the comparison is not valid if one or both of the states 
possess a number of attributes that are not included in the scales. It was 
for this reason that we listed in Chapter 3 all the attributes that were not 
included in the primary scales. Anybody seeking to use the scales should be 



ERJ.C 



5-6 



aware of those attributes that were not included and temper any interpreta- 
tions accordingly. 

The third caveat concerns the degree to which each scale can be inter- 
preted as reflecting a developmental pattern. Up to this point in the dis- 
cussion, the only assumption that has been made about the scales is that they 
are hierarchical. Given that the scales are hierarchical and that this 
hierarchy is invariant across both states and points in time, it is appro- 
priate to use the scales to assess differences between the states and to 
assess growth. Should the scales not only be hierarchical but also develop- 
mental, then their interpretive power would be enhanced considerably. In 
particular* knowledge about where a state is with respect to each of the 
scales could be used to establish a set of expectations regarding the 
continued development of the state's dissemination system. Put another way, 
the indicants in each scale could serve as a series of "mileposts" which 
would enable us to chart a state's progress in building dissemination capa- 
city. 

In Chapter 4, however, we found only occasional evidence to support the 
interpretation of the scales as being developmental as well as hierarchical. 
The preceeding statement needs to be tempered, however, by an awareness of 
the extremely limited amount of data that was available for these analyses. 
Since information WuS collected only in 1978 and 1979, we were unable to 
chart the growth of the Cohort I and Cohort II states during the initial 
three and two years, respectively, of their participation in the SCBP pro- 
gram. Since the states in the first two Cohorts constituted the bulk of our 
sample, only minimal information was available regarding dissemination system 
growth during the initial phase of SCBP participation. With so few observa- 
tions, the precision with which we could describe the relationship between 
scale position and years of participation wds substantially impaired. 
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Clearly, when a point in the graphs presented in Figures 4,1 - 4.6 represents 
only two or three states, the ;)OSition of that point is far from stable. Even 
small variations with respect to just one state's score would cause that point 
to shift position. Furthermore, the data analyses in Chapter 4 suggested that 
much of the dissemination system's growth might occur during the first two 
years and that, by years three and four, growth tends to level off. 

In addition to the difficulties associated with trying to describe the 
growth of dissemination systems during the first few years of SCBP participa- 
tion, our ability to observe and describe dissemination system growth is fur- 
ther impaired by the overpowering impact that contextual characteristics, such 
as political constraints, can have on the development of a dissemination 
system. Each of the states in our sample is unique in so many ways that it is 
extremely difficult to perceive anything that might be common to all states. 
This problem, which tends to be present in all social science research, is 
that the variations between individuals or groups of individuals (such as 
states) tend to be so large that patterns of behavior cciinon to all indivi- 
duals are obscured. The usual approach to dealing with such a problem is to 
increase the number of observations, which, of course, was not an option 
available to us in this study. Consequently, the state and SEA characteris- 
tics, especially those characteristics that led certain states to be early 
participants in the SCBP program, may have completely obscured the relation- 
ship between dissemination system development and years of participation in 
the SCBP program. 

Thus, while we were unable to find much support in the available data 
for a developmental interpretation of the scales, it must be emphasized that 
this does not rule out the possibility that the scales represent a development 
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methaphor. It is quite likely, given the limited afnount of available data, 
that the developmental metaphor could have been obscured by the absence of 
sufficient information describing how dissemination systems grow in the early 
years of their inception and by the unique characteristics of the states. 

Having now considered how the scales can be used to assess differences 
and to assess growth, and having also considered the three major caveats that 
must be considered in interpreting and utilizing scale scores, we turn to a 
discussion of two applications for which we believe the scales are particu- 
larly well suited. The first major use of the scale scores would be to per- 
mit those associated with a state's dissemination system, as well as those 
external to it, to assess the system's status. In particular, the scales can 
be used to establish contexts against which the status of the state's dis- 
semination system configuration can be assessed. 

The most obvious of these contexts is one which permits the state to be 
compared to the dissemination system configurations of all of the other 
states in this study. This would be what educational researchers refer to as 
"norm- referenced" interpretation, in which the status of a dissemination sys- 
t'';m relative to dissemination systems throughout the country would be 
assessed. Such an assessment could be used to indicate aspects of the dis- 
semination system where further development is needed, as well as those 
aspects in which a sufficiently high level has already been achieved. Such 
an assessment, of course, would need to be tempered by an awareness of 
v/hether the dissemination system was still in a fledgling state or more ad- 
vanced stage of development, as well as a realization that certain components 
of the dissemination system might not be reflected in the scale scores. In 
addition, such comparisons would have to be tempered by the unique political, 
demographic, and educationa considerations that may exist in a state. 



Taking all of these considerations into account, however, a norm-referenced 
interpretation could be useful in indicating what aspects of a dissemination 
system might be targeted for further development. 

This same information could also be used by NIE in assessing the pro- 
posed emphases contained in the SCBP applications. Given a profile of a 
state's dissemination system with respect to each of the scales, a state 
which proposes to continue to spend SCBP funds on development of its resource 
base and yet which already ranks well above the national average with respect 
to its resource base might be questioned as to the appropriateness of such a 
priority. Of course, the political, demographic, and educational constraints 
under which the state must operate should all be taken into av^count in making 
such a judgment. For instance, a large state with a sparse population might 
find it much more appropriate to invest in the continued development than 
might a small sta^e with a very dense population. 

To summarize, then, the scales can be used by both the states and NIE to 
assess the status of a dissemination system configuration relative to the 
dissemination systems of other states. While such an assessment must take 
into account certain considerations that will be unique to a state, nonethe- 
less, the scales can provide some guidelines as to where additional develop- 
ment might be needed or not needed. Clearly, the usefulness of the scales 
will be enhanced with the inclusion of more information in the decision- 
making process, such as the scores of the dissemination system at some pre- 
vious time. 

The second major use of the scale scores would be to measure change as 
well as status. Furthermore, change can be measured not just over two points 
in time, but across several points in time. From the state perspective, the 
ability to assess change using the scales has several important applications. 
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To begin with, objectives can be stated at the bey.nning of the year with 
regard to how the dissemination system configuration will be modified, and 
the degree to which these objectives have been met can then be assessed at 
the end of the year by comparing the change in the scale scores. For in- 
stance, if a state has as one of its objectives increasing the accessibility 
of the dissemination system to users, the amount of change in the score on 
the coordination of the resource base scale can be used, at least partially, 
to assess the extent to which this objective has been met. By the same 
token, changes with respect to the institutionalization scale can be used to 
partially assess the degree to which the dissemination system functions are 
being underwritten by the state as opposed to requiring federal support. If 
data on the scales are available for several years, then this retrospective 
information could potentially be used to establish objectives and expecta- 
tions about what areas or aspects of the dissemination system should be 
targeted for further development. 

As might be expected, virtually the same sorts of uses apply to NIE as 
well. In the course of monitoring the SCBP project, if the project applica- 
tion targets some aspect of the dissemination system for further development 
and no change occurs with respect to the appropriate scale score, then this 
might be taken as an indication by NIE staff that the operation of the pro- 
gram with respect to the objective should be scrutinized further. It may 
well be that the political, demographic, and educational constraints in that 
state acted in some fashion to render mt-'eting the objective impossible. 
Nonetheless, the scale scores do provide a convenient and concise way of 
identifying areas where the states are having difficulty meeting their 
objectives. In addition, if multiple data points are available, the NIE 
staff can take a retrospective view of dissemination system development in a 
state and based ipon that review have certain expectations about what sorts 
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of emphases should be in the next project application. We are not suggesting 
that these expectations should necessarily be based on some sort of develop- 
mental model, sine" as was stated in a previous chapter, no evidence has yet 
been found to support the evidence of any sort of developmental paradigm. 
However, NIE monitors who are familiar with the states and their unique 
characteristics could combine that knowledge with the profile and the know- 
ledge of how the SCBP program has been operating to arrive at some expecta- 
tions regarding what future directions the SCBP program should take. 

SUMMARY 

Even with the constraints that the caveats discussed in this chapter 
place upon their use, the scales can be a powerful interpretative tool in 
assessing a state's dissemination system configuration. We do not believe, 
however, that the construction and application of scales such as those that 
have been described in this substudy are necessarily limited to dissemination 
system configurations. Given that the relevent attributes underlying an 
organization can be identified and described, then it is likely that a set of 
indicants can be generated which will describe each of the attributes and a 
scale can be constructed. As this study has shown, it is not unreasonable to 
expect that many of the indicants will form a hierarchy, and hence can be 
legitimately considered to constitute a scale. Such a scale can be used to 
not only provide a succinct description of an organization, but can also be 
used as a basis for comparing organizations. Should the hierarchy of the 
indicants remain invariant across repeated administrations of the indicants, 
then the scale can be used to measure organizational growth. Finally, if 
that organizational growth can be shown to follow a consistent pattern, then 
the scale can be used to not only measure growth to date, but also to 
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establish expectations regarding future growth. While this last possibility 
was not present in the dissemination system scales, the usefulness of the 
scales in making comparisons and measuring growth is capable of providing 
individuals both w ^nin and external to the organization with a powerful tool 
in assessing the status of their organization. Such assessments can be used 
as a valuable source of input to the decision- maki.g p,ocess regarding what 
sorts of activities related to the organization should oe undertaken or 
terminated. 



ERIC 



5-13 ^'^'J 



REFERENCES 



The NTS Research Corporation reports for this study all have the general 
title Building Capacity for the Improvement of Educational Prajtice . The 
complete set of volume is as follows: 



Volume I Final Report: Building Capacity for the Improvement 
of Education, An Evaluation of NIE's State Dissemi- 
nation Grants Program (April 1981) 

Volume II 1979 State Abstracts: State Dissemination Efforts 
(April 1980) 

Volume III A Study of Linker Agent Activities and Roles 
(April 1981) 

Volume IV A Study of the Development of Scales Measuring 
Dissemination Capacity (April 1981) 

Volume V Executive Summary 
(April 1981) 



For further information contact: 



NTS RESEARCH CORPORATION 



2634 Chapel Hill Boulevard 
Durham, NC 27707 
(919) 493-.?451 



Eugene C. Royster 
Principal Investigator 

Doren L. Madey 
Project Director 



NATIONAL INSTITUTE OF EDUCATION 

Program on Disseiiination and 
liTiprovement of Practice 
Research and Educational 
Practice Unit 
(202) 254-6050 

Michael B. Kane 

NIE Assistant Director 

John C. Egermeier 
Project Officb;' 



